Jul 10 18:37:35.705161 ip-10-0-140-163 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 10 18:37:35.705172 ip-10-0-140-163 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 10 18:37:35.705179 ip-10-0-140-163 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 10 18:37:35.705393 ip-10-0-140-163 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 10 18:37:44.975447 ip-10-0-140-163 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 10 18:37:44.975463 ip-10-0-140-163 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bcb2c7cb8bac448481e306f7cf7c919b --
Jul 10 18:39:12.940084 ip-10-0-140-163 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.394447    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398908    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398921    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398926    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398929    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398933    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398936    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398939    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:13.432000 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398942    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398945    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398948    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398951    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398954    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398957    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398959    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398962    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398965    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398968    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398970    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398973    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398975    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398980    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398989    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398993    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398995    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.398998    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399001    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399003    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:13.434730 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399006    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399009    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399012    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399015    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399018    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399021    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399026    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399029    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399033    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399036    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399039    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399042    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399045    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399047    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399050    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399053    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399056    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399058    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399061    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:13.438089 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399064    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399066    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399069    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399072    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399074    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399077    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399080    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399082    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399086    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399088    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399091    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399093    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399096    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399099    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399101    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399104    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399107    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399109    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399112    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399115    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:13.441113 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399117    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399120    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399123    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399126    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399129    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399131    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399134    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399137    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399140    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399143    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399145    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399148    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399152    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399154    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399157    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399159    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399161    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399164    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399168    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:13.444484 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399171    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399174    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399178    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399181    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399184    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399187    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399190    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399193    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399195    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399198    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399200    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399203    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399206    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399209    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.399991    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400003    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400006    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400010    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400013    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400017    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:13.447420 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400019    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400022    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400025    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400028    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400031    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400034    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400037    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400039    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400042    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400044    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400048    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400050    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400053    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400057    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400059    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400062    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400065    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400069    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400074    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:13.451382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400076    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400079    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400082    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400085    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400088    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400091    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400094    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400096    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400099    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400102    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400104    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400107    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400110    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400112    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400115    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400117    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400120    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400122    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400125    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400127    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:13.454467 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400130    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400133    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400136    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400139    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400141    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400144    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400147    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400150    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400152    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400155    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400159    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400163    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400166    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400169    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400171    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400175    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400178    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400181    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400183    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400186    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:13.458549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400189    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400194    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400198    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400201    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400204    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400207    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400209    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400212    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400214    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400217    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400219    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400222    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400224    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400227    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400229    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400233    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400235    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400238    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400240    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400243    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:13.461382 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400246    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400248    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400251    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400256    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400259    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400261    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400264    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400266    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400269    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400272    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400274    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400277    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400279    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400282    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400642    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400647    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400649    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400652    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400655    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400658    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:13.465161 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400661    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400664    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400666    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400669    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400671    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400674    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400677    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400679    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400682    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400685    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400687    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400690    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400692    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400695    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400698    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400702    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400704    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400707    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400710    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:13.468082 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400712    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400714    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400717    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400721    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400723    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400726    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400728    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400731    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400734    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400736    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400739    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400741    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400743    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400746    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400749    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400751    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400754    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400756    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400777    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400780    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:13.471873 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400782    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400785    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400788    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400790    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400793    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400796    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400799    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400802    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400806    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400809    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400812    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400814    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400817    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400819    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400822    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400829    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400833    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400839    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400842    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:13.475021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400845    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400848    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400851    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400854    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400857    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400859    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400862    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400864    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400867    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400870    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400872    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400876    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400879    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400882    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400884    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400887    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400890    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400892    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400895    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400898    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:13.478691 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400900    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400904    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400907    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400909    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400912    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400914    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400917    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400919    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400922    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400925    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400928    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400930    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400934    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400938    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.400941    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401012    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401022    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401031    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401037    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401043    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401047    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 18:39:13.481549 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401052    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 18:39:13.480215 ip-10-0-140-163 systemd[1]: Started Kubernetes Kubelet.
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401056    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401062    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401065    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401068    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401072    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401075    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401078    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401081    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401084    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401087    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401090    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401097    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401100    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401103    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401106    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401110    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401113    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401116    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401119    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401122    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401126    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401129    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401132    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 18:39:13.486148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401135    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401138    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401142    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401145    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401147    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401150    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401153    2558 flags.go:64] FLAG: --enable-server="true"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401156    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401161    2558 flags.go:64] FLAG: --event-burst="100"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401164    2558 flags.go:64] FLAG: --event-qps="50"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401168    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401172    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401175    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401178    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401181    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401184    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401187    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401190    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401193    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401196    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401200    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401203    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401206    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401208    2558 flags.go:64] FLAG: --feature-gates=""
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401221    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 18:39:13.490111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401224    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401227    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401230    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401233    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401236    2558 flags.go:64] FLAG: --help="false"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401240    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401243    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401246    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401249    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401252    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401255    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401258    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401261    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401264    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401267    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401270    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401273    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401277    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401280    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401283    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401286    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401289    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401292    2558 flags.go:64] FLAG: --lock-file=""
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401294    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 18:39:13.496007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401297    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401301    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401306    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401310    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401313    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401316    2558 flags.go:64] FLAG: --logging-format="text"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401319    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401322    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401325    2558 flags.go:64] FLAG: --manifest-url=""
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401327    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401332    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401335    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401339    2558 flags.go:64] FLAG: --max-pods="110"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401342    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401345    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401348    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401351    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401354    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401357    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401360    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401367    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401370    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401373    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 18:39:13.522443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401376    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401379    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401385    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401388    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401391    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401394    2558 flags.go:64] FLAG: --port="10250"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401397    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401400    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05b827bcb6d8a65d7"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401403    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401406    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401409    2558 flags.go:64] FLAG: --register-node="true"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401412    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401415    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401420    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401423    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401426    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401429    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401432    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401435    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401438    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401441    2558 flags.go:64] FLAG: --runonce="false"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401444    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401447    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401450    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401453    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 18:39:13.540933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401456    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401459    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401462    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401515    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401520    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401523    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401526    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401530    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401533    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401537    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401540    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401546    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401548    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401551    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401555    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401558    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401561    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401564    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401567    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401570    2558 flags.go:64] FLAG: --v="2"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401575    2558 flags.go:64] FLAG: --version="false"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401579    2558 flags.go:64] FLAG: --vmodule=""
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401583    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.401586    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401697    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:13.701741 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401701    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401704    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401708    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401713    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401716    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401719    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401721    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401725    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401727    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401730    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401733    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401736    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401738    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401741    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401743    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401746    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401750    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401753    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401756    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401773    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:13.702835 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401776    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401779    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401781    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401784    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401787    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401789    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401792    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401797    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401800    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401803    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401805    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401808    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401811    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401813    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401816    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401819    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401821    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401824    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401826    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401829    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:13.703611 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401832    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401835    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401837    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401840    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401842    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401845    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401848    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401850    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401854    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401856    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401859    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401861    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401864    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401867    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401869    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401872    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401874    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401878    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401882    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:13.704332 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401886    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401889    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401892    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401895    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401898    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401901    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401904    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401906    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401909    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401912    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401914    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401917    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401920    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401922    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401925    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401928    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401930    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401933    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401935    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401938    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:13.704975 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401940    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401946    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401949    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401952    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401955    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401958    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401960    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401963    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401966    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401968    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401971    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401974    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401978    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401981    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401984    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401986    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401989    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401992    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:13.705665 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.401994    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.402843    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.414370    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.414387    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.414407    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.414418    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414475    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414479    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414483    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414486    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414489    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414492    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414494    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414497    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:13.706375 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414500    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414503    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414506    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414509    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414511    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414514    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414517    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414520    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414523    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414526    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414529    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414532    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414534    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414537    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414540    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414545    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414548    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414551    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414553    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:13.706996 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414556    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414559    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414561    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414564    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414567    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414569    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414572    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414574    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414577    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414580    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414582    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414585    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414588    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414591    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414594    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414596    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414599    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414602    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414604    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414608    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:13.707659 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414610    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414613    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414616    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414618    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414621    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414624    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414626    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414629    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414634    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414638    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414641    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414644    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414647    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414650    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414652    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414655    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414658    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414661    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414663    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414666    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:13.710898 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414668    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414671    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414674    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414676    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414680    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414683    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414686    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414688    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414691    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414695    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414698    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414701    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414704    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414707    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414709    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414712    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414715    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414717    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414720    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:13.715294 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414723    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414727    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414732    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414735    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414738    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414741    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414744    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414748    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414751    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414754    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414756    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414776    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414780    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.414785    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414910    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 18:39:13.720251 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414915    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414918    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414920    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414925    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414928    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414931    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414933    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414936    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414938    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414941    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414944    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414947    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414950    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414953    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414956    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414958    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414961    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414964    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414968    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414971    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 18:39:13.722358 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414973    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414976    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414978    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414981    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414984    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414986    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414989    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414991    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414994    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414997    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.414999    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415002    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415004    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415007    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415010    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415013    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415016    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415018    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415021    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 18:39:13.725830 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415023    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415026    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415030    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415033    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415036    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415043    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415046    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415049    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415051    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415054    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415057    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415060    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415063    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415066    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415070    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415073    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415076    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415079    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415082    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 18:39:13.726419 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415085    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415087    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415090    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415093    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415095    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415098    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415100    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415103    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415107    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415112    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415115    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415117    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415120    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415123    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415126    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415129    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415132    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415135    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415138    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415141    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 18:39:13.727230 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415144    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415147    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415149    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415152    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415157    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415159    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415162    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415164    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415167    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415169    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415172    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415174    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415177    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415179    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415182    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415185    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415187    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415190    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415192    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 18:39:13.728069 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:13.415195    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.415199    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.415300    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.418674    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.420141    2558 server.go:1019] "Starting client certificate rotation"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.422450    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.422487    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.427460    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.432068    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.444980    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.448214    2558 log.go:25] "Validated CRI v1 image API"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.449105    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 18:39:13.728695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.452306    2558 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a455bc51-3761-40d1-9c61-8791221e1a0d:/dev/nvme0n1p3 d0df85e6-de2a-473a-9716-19bfa1a2294e:/dev/nvme0n1p4]
Jul 10 18:39:13.729096 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.452320    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.456801    2558 manager.go:216] Machine: {Timestamp:2026-07-10 18:39:13.455436694 +0000 UTC m=+0.383082033 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088303 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2af396e9ef39015250cce6cf29f7e5 SystemUUID:ec2af396-e9ef-3901-5250-cce6cf29f7e5 BootID:bcb2c7cb-8bac-4484-81e3-06f7cf7c919b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a7:b6:8c:98:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a7:b6:8c:98:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:ff:44:10:c4:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.456904    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.456975    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.460977    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.461008    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-163.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.461948    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.461967    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 18:39:13.729258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.461979    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.461997    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.463460    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.466923    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.467054    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.467086    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.467098    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.467117    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.467131    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.467142    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.468418    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.469556    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.469613    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.472221    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474393    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474420    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474584    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474596    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474602    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474607    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474616    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.474622    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.475300    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.475307    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.475316    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.475323    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.475331    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.475339    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.476582    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.476589    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.479494    2558 server.go:1268] "Started kubelet"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.479732    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.479754    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.479858    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.481286    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.483195    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.483289    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-163.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.483326    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.483385    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 18:39:13.730051 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.486757    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.489264    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-163.ec2.internal.18c101a929e60f76  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-163.ec2.internal,UID:ip-10-0-140-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-163.ec2.internal,},FirstTimestamp:2026-07-10 18:39:13.479470966 +0000 UTC m=+0.407116305,LastTimestamp:2026-07-10 18:39:13.479470966 +0000 UTC m=+0.407116305,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-163.ec2.internal,}"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.491089    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.491107    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.491164    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.491984    2558 factory.go:55] Registering systemd factory
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492022    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492200    2558 factory.go:153] Registering CRI-O factory
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492213    2558 factory.go:223] Registration of the crio container factory successfully
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492297    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492320    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492335    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492324    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492404    2558 factory.go:103] Registering Raw factory
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492420    2558 manager.go:1196] Started watching for new ooms in manager
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.492871    2558 manager.go:349] Starting recovery of all containers
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.493254    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.493264    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.493478    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.493548    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-163.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.504210    2558 manager.go:354] Recovery completed
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.522679    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.523708    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-22jxv"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.524575    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.524604    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.524618    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.525259    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.525267    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.525274    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.525281    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.525967    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.525991    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.526001    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:13.731242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.526661    2558 policy_none.go:49] "None policy: Start"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.526676    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.526688    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.527589    2558 policy_none.go:47] "Start"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.531502    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-22jxv"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.535756    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-163.ec2.internal.18c101a92c967760  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-163.ec2.internal,UID:ip-10-0-140-163.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-163.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-163.ec2.internal,},FirstTimestamp:2026-07-10 18:39:13.524586336 +0000 UTC m=+0.452231675,LastTimestamp:2026-07-10 18:39:13.524586336 +0000 UTC m=+0.452231675,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-163.ec2.internal,}"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.575329    2558 manager.go:337] "Starting Device Plugin manager"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.575354    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.575363    2558 server.go:87] "Starting device plugin registration server"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.575568    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.575580    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.575725    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.575868    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.575878    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.576389    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.576439    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.584481    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.585578    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.585610    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.585627    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.585655    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.587903    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.676330    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.677687    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.677716    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.677727    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.677753    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.686204    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"]
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.686267    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.687103    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.687125    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-163.ec2.internal\": node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.687127    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:13.873372 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.687211    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.687226    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689090    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689232    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689265    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689909    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689941    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689947    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689955    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689975    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.689988    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.692172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.692192    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.693359    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.693383    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.693392    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.694660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b171dc9bc69a4139a3473bd963deb50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"8b171dc9bc69a4139a3473bd963deb50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.694681    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b171dc9bc69a4139a3473bd963deb50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"8b171dc9bc69a4139a3473bd963deb50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.694697    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4084b00d8aece702c3f7f6e211f96825-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.ec2.internal\" (UID: \"4084b00d8aece702c3f7f6e211f96825\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.704315    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.765547    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.ec2.internal\" not found" node="ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.769957    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.ec2.internal\" not found" node="ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.795259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b171dc9bc69a4139a3473bd963deb50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"8b171dc9bc69a4139a3473bd963deb50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.874400 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.795426    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b171dc9bc69a4139a3473bd963deb50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"8b171dc9bc69a4139a3473bd963deb50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.875035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.795443    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4084b00d8aece702c3f7f6e211f96825-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.ec2.internal\" (UID: \"4084b00d8aece702c3f7f6e211f96825\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.875035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.795465    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4084b00d8aece702c3f7f6e211f96825-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.ec2.internal\" (UID: \"4084b00d8aece702c3f7f6e211f96825\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.875035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.795354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b171dc9bc69a4139a3473bd963deb50-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"8b171dc9bc69a4139a3473bd963deb50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.875035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:13.795515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b171dc9bc69a4139a3473bd963deb50-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal\" (UID: \"8b171dc9bc69a4139a3473bd963deb50\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:13.875035 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.804802    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:13.991253 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:13.905400    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:14.053263 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:14.006238    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:14.193713 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:14.068449    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:14.193713 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:14.073907    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:14.193713 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:14.106462    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:14.344364 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:14.210081    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:14.344364 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:14.310625    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:16.165029 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:14.410672    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:16.165029 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:14.420702    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 18:39:16.165029 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:14.420886    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 18:39:16.293324 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:16.177112    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:16.293324 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.201254    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:34:15 +0000 UTC" deadline="2028-02-10 20:50:31.204565989 +0000 UTC"
Jul 10 18:39:16.293324 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.201275    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13922h11m15.003294075s"
Jul 10 18:39:16.293324 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:16.277752    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-163.ec2.internal\" not found"
Jul 10 18:39:16.296485 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.295864    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 18:39:16.299559 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.298920    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 18:39:16.299559 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.298946    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 18:39:16.299559 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.298959    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeHasSufficientPID"
Jul 10 18:39:16.419725 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.343690    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:16.419725 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.357556    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:16.419725 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.372136    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:39:16.419725 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.375078    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal"
Jul 10 18:39:16.419725 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.384537    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 18:39:16.526944 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:16.437482    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b171dc9bc69a4139a3473bd963deb50.slice/crio-b22d2fe842894d5d6834877859b4920a581b9187c86d48495c52dee8ab194fe9 WatchSource:0}: Error finding container b22d2fe842894d5d6834877859b4920a581b9187c86d48495c52dee8ab194fe9: Status 404 returned error can't find the container with id b22d2fe842894d5d6834877859b4920a581b9187c86d48495c52dee8ab194fe9
Jul 10 18:39:16.526944 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.441581    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:39:16.641131 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.549924    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:16.803258 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:16.701380    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4084b00d8aece702c3f7f6e211f96825.slice/crio-f1904861d1124d727a9d62459f563091abf006ef4493591308b313c9bcdae339 WatchSource:0}: Error finding container f1904861d1124d727a9d62459f563091abf006ef4493591308b313c9bcdae339: Status 404 returned error can't find the container with id f1904861d1124d727a9d62459f563091abf006ef4493591308b313c9bcdae339
Jul 10 18:39:16.803258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.738126    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:17.036905 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:16.975863    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.134783    2558 apiserver.go:52] "Watching apiserver"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.143095    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.143505    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-sqq97","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7","openshift-cluster-node-tuning-operator/tuned-xh9z9","openshift-image-registry/node-ca-jqvjm","openshift-multus/multus-additional-cni-plugins-2dpm8","openshift-network-diagnostics/network-check-target-p7nlk","openshift-network-operator/iptables-alerter-8rgnm","openshift-ovn-kubernetes/ovnkube-node-4l7jt","kube-system/global-pull-secret-syncer-bcvk5","kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal","openshift-dns/node-resolver-g5kd9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal","openshift-multus/multus-xbw6g","openshift-multus/network-metrics-daemon-j7qn6"]
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.144966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.146258    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.147371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.147598    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.147801    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.147834    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gkqqh\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.148099    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.148383    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.148441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.148487    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.149174    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.149290    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9hb7r\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.149356    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.149405    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.149628    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.149665    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nnp7c\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.149686    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.150425    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.150728    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.150728    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.150790    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bd2qc\""
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.150813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.150872    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:17.190128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.151812    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.151869    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.151955    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.151955    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b56pt\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.152024    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.153008    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.154276    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.155179    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.155223    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.155299    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z9qh7\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.155584    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.155631    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.156440    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zfwfb\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.156734    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.156993    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.157253    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.157610    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.157711    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.159868    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xvwvh\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.160119    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.160225    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.160221    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.162543    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.162954    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c8mhw\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.163236    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.163491    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.163513    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.163537    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 18:39:17.192005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.163566    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170570    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-registration-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170587    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-cni-multus\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170620    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-conf-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170639    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-kubernetes\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170661    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lc58\" (UniqueName: \"kubernetes.io/projected/7b0bfc28-5d5c-42b3-aada-e8f37187da96-kube-api-access-7lc58\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-cnibin\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170791    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-k8s-cni-cncf-io\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-multus-certs\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170879    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170921    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-sys-fs\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovn-node-metrics-cert\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-cni-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.195471 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.170999    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24qgs\" (UniqueName: \"kubernetes.io/projected/ddf22656-c128-49dc-80fb-94a983d5ebad-kube-api-access-24qgs\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-var-lib-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d494c4a-3c15-495c-a993-9bbf592b6d50-kubelet-config\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-slash\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171137    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-hostroot\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171160    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-os-release\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171210    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d494c4a-3c15-495c-a993-9bbf592b6d50-dbus\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171231    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-run-netns\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171254    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-system-cni-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddf22656-c128-49dc-80fb-94a983d5ebad-cni-binary-copy\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-daemon-config\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171394    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fae3b187-d48f-43e9-a6a3-8bd2dae78715-konnectivity-ca\") pod \"konnectivity-agent-sqq97\" (UID: \"fae3b187-d48f-43e9-a6a3-8bd2dae78715\") " pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.196211 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171423    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-lib-modules\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171469    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-tmp-dir\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-env-overrides\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-cni-bin\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171590    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd859782-385f-4f5b-96cd-f10e405bfdbb-tmp\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171606    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-socket-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171650    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-kubelet\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171711    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-netns\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171756    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-kubelet\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171784    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-hosts-file\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171799    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-node-log\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znr5q\" (UniqueName: \"kubernetes.io/projected/fcc1787d-a277-48fe-b6ea-8dde75b794ef-kube-api-access-znr5q\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171838    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.196972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-cni-netd\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171895    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovnkube-script-lib\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-os-release\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-systemd\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.171998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzqn4\" (UniqueName: \"kubernetes.io/projected/cd859782-385f-4f5b-96cd-f10e405bfdbb-kube-api-access-bzqn4\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172041    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovnkube-config\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-iptables-alerter-script\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172079    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysctl-conf\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-host\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172116    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc1787d-a277-48fe-b6ea-8dde75b794ef-host\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172202    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.197786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-etc-kubernetes\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172255    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-ovn\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172288    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-tuned\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cnibin\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysctl-d\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-var-lib-kubelet\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172389    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnf4d\" (UniqueName: \"kubernetes.io/projected/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-kube-api-access-bnf4d\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9xc\" (UniqueName: \"kubernetes.io/projected/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-kube-api-access-dg9xc\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172450    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-log-socket\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-system-cni-dir\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172502    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fae3b187-d48f-43e9-a6a3-8bd2dae78715-agent-certs\") pod \"konnectivity-agent-sqq97\" (UID: \"fae3b187-d48f-43e9-a6a3-8bd2dae78715\") " pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172523    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-modprobe-d\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-systemd\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.198570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25gjv\" (UniqueName: \"kubernetes.io/projected/86ec85bc-771c-42e6-8c4b-6a31203c35c9-kube-api-access-25gjv\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172589    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnlzx\" (UniqueName: \"kubernetes.io/projected/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-kube-api-access-jnlzx\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-cni-bin\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172656    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-host-slash\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-etc-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtc8f\" (UniqueName: \"kubernetes.io/projected/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-kube-api-access-mtc8f\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172831    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysconfig\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-sys\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcc1787d-a277-48fe-b6ea-8dde75b794ef-serviceca\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-device-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172923    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-systemd-units\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-socket-dir-parent\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.278673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.172998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.173018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-run\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.188824    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-clwtf"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.196302    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-clwtf"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.259012    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.260519    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" event={"ID":"4084b00d8aece702c3f7f6e211f96825","Type":"ContainerStarted","Data":"f1904861d1124d727a9d62459f563091abf006ef4493591308b313c9bcdae339"}
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.261990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" event={"ID":"8b171dc9bc69a4139a3473bd963deb50","Type":"ContainerStarted","Data":"b22d2fe842894d5d6834877859b4920a581b9187c86d48495c52dee8ab194fe9"}
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273308    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-cni-bin\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-host-slash\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273361    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-etc-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273411    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-etc-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273417    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-cni-bin\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-host-slash\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273488    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtc8f\" (UniqueName: \"kubernetes.io/projected/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-kube-api-access-mtc8f\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.279437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysconfig\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-sys\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-sys\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysconfig\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273646    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcc1787d-a277-48fe-b6ea-8dde75b794ef-serviceca\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-device-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273685    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-systemd-units\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273702    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-socket-dir-parent\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273761    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-systemd-units\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-device-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273790    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273813    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-run\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-registration-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273867    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-socket-dir-parent\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.280266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273909    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-cni-multus\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-run\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274062    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.273869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-cni-multus\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-conf-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-kubernetes\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274131    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274152    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lc58\" (UniqueName: \"kubernetes.io/projected/7b0bfc28-5d5c-42b3-aada-e8f37187da96-kube-api-access-7lc58\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-cnibin\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-registration-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-k8s-cni-cncf-io\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-conf-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274225    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-multus-certs\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274244    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcc1787d-a277-48fe-b6ea-8dde75b794ef-serviceca\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:17.281017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-cnibin\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274274    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-kubernetes\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-sys-fs\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovn-node-metrics-cert\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274316    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-multus-certs\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-cni-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24qgs\" (UniqueName: \"kubernetes.io/projected/ddf22656-c128-49dc-80fb-94a983d5ebad-kube-api-access-24qgs\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-var-lib-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d494c4a-3c15-495c-a993-9bbf592b6d50-kubelet-config\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-slash\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-sys-fs\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-cni-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274305    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-k8s-cni-cncf-io\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.281480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-hostroot\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-os-release\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-var-lib-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274536    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274536    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d494c4a-3c15-495c-a993-9bbf592b6d50-dbus\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274569    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2d494c4a-3c15-495c-a993-9bbf592b6d50-kubelet-config\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274603    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-slash\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2d494c4a-3c15-495c-a993-9bbf592b6d50-dbus\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-run-netns\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-hostroot\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274696    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-os-release\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-system-cni-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274718    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-run-netns\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274757    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-system-cni-dir\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddf22656-c128-49dc-80fb-94a983d5ebad-cni-binary-copy\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274785    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-daemon-config\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274790    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fae3b187-d48f-43e9-a6a3-8bd2dae78715-konnectivity-ca\") pod \"konnectivity-agent-sqq97\" (UID: \"fae3b187-d48f-43e9-a6a3-8bd2dae78715\") " pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274817    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-lib-modules\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-tmp-dir\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274855    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-env-overrides\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-cni-bin\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274885    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd859782-385f-4f5b-96cd-f10e405bfdbb-tmp\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-socket-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274918    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274936    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-kubelet\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-netns\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-kubelet\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274986    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.282877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.274997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-hosts-file\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-node-log\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-hosts-file\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znr5q\" (UniqueName: \"kubernetes.io/projected/fcc1787d-a277-48fe-b6ea-8dde75b794ef-kube-api-access-znr5q\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275164    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-cni-netd\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovnkube-script-lib\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-os-release\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-systemd\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-tmp-dir\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275265    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzqn4\" (UniqueName: \"kubernetes.io/projected/cd859782-385f-4f5b-96cd-f10e405bfdbb-kube-api-access-bzqn4\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275296    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-cni-bin\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275339    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-node-log\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovnkube-config\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283450 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275337    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddf22656-c128-49dc-80fb-94a983d5ebad-cni-binary-copy\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275362    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddf22656-c128-49dc-80fb-94a983d5ebad-multus-daemon-config\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-iptables-alerter-script\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275415    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-kubelet\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275457    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-var-lib-kubelet\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275509    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-socket-dir\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275549    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275552    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysctl-conf\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysctl-conf\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275695    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-env-overrides\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-host-run-netns\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275788    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-host\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275797    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-systemd\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-os-release\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.283948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc1787d-a277-48fe-b6ea-8dde75b794ef-host\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275826    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-iptables-alerter-script\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-lib-modules\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275924    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275934    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-host-cni-netd\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-host\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275960    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovnkube-config\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcc1787d-a277-48fe-b6ea-8dde75b794ef-host\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.275992    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.275996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276025    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-etc-kubernetes\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-ovn\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.276054    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:17.284564 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.276070    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret podName:2d494c4a-3c15-495c-a993-9bbf592b6d50 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:17.776049739 +0000 UTC m=+3.037714515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret") pod "global-pull-secret-syncer-bcvk5" (UID: "2d494c4a-3c15-495c-a993-9bbf592b6d50") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276087    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-openvswitch\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-ovn\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276161    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-tuned\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276169    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddf22656-c128-49dc-80fb-94a983d5ebad-etc-kubernetes\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.276213    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:17.776190754 +0000 UTC m=+3.037855509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276232    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cnibin\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysctl-d\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276310    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-var-lib-kubelet\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276335    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnf4d\" (UniqueName: \"kubernetes.io/projected/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-kube-api-access-bnf4d\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-cnibin\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9xc\" (UniqueName: \"kubernetes.io/projected/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-kube-api-access-dg9xc\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-log-socket\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276446    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-var-lib-kubelet\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276447    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovnkube-script-lib\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-system-cni-dir\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-log-socket\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fae3b187-d48f-43e9-a6a3-8bd2dae78715-agent-certs\") pod \"konnectivity-agent-sqq97\" (UID: \"fae3b187-d48f-43e9-a6a3-8bd2dae78715\") " pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-modprobe-d\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-sysctl-d\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276582    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-systemd\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276606    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25gjv\" (UniqueName: \"kubernetes.io/projected/86ec85bc-771c-42e6-8c4b-6a31203c35c9-kube-api-access-25gjv\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnlzx\" (UniqueName: \"kubernetes.io/projected/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-kube-api-access-jnlzx\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-system-cni-dir\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-modprobe-d\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.276758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86ec85bc-771c-42e6-8c4b-6a31203c35c9-run-systemd\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.277375    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fae3b187-d48f-43e9-a6a3-8bd2dae78715-konnectivity-ca\") pod \"konnectivity-agent-sqq97\" (UID: \"fae3b187-d48f-43e9-a6a3-8bd2dae78715\") " pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.278943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86ec85bc-771c-42e6-8c4b-6a31203c35c9-ovn-node-metrics-cert\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.279284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cd859782-385f-4f5b-96cd-f10e405bfdbb-etc-tuned\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.285890 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.279315    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd859782-385f-4f5b-96cd-f10e405bfdbb-tmp\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.286642 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.279450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fae3b187-d48f-43e9-a6a3-8bd2dae78715-agent-certs\") pod \"konnectivity-agent-sqq97\" (UID: \"fae3b187-d48f-43e9-a6a3-8bd2dae78715\") " pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.286642 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.284022    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:17.286642 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.284043    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:17.286642 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.284056    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9njsl for pod openshift-network-diagnostics/network-check-target-p7nlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:17.286642 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.284136    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl podName:5a7bf1fa-bfba-4190-be81-f7bada15234e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:17.784118529 +0000 UTC m=+3.045783299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9njsl" (UniqueName: "kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl") pod "network-check-target-p7nlk" (UID: "5a7bf1fa-bfba-4190-be81-f7bada15234e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:17.286642 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.286636    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24qgs\" (UniqueName: \"kubernetes.io/projected/ddf22656-c128-49dc-80fb-94a983d5ebad-kube-api-access-24qgs\") pod \"multus-xbw6g\" (UID: \"ddf22656-c128-49dc-80fb-94a983d5ebad\") " pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.292956 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.292500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnf4d\" (UniqueName: \"kubernetes.io/projected/db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f-kube-api-access-bnf4d\") pod \"aws-ebs-csi-driver-node-jk5b7\" (UID: \"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.292956 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.292544    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzqn4\" (UniqueName: \"kubernetes.io/projected/cd859782-385f-4f5b-96cd-f10e405bfdbb-kube-api-access-bzqn4\") pod \"tuned-xh9z9\" (UID: \"cd859782-385f-4f5b-96cd-f10e405bfdbb\") " pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.293139 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.293119    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25gjv\" (UniqueName: \"kubernetes.io/projected/86ec85bc-771c-42e6-8c4b-6a31203c35c9-kube-api-access-25gjv\") pod \"ovnkube-node-4l7jt\" (UID: \"86ec85bc-771c-42e6-8c4b-6a31203c35c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.293194 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.293178    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9xc\" (UniqueName: \"kubernetes.io/projected/88cc0f38-18da-4e79-b96f-5b6435cdf0d7-kube-api-access-dg9xc\") pod \"node-resolver-g5kd9\" (UID: \"88cc0f38-18da-4e79-b96f-5b6435cdf0d7\") " pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.293732 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.293708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znr5q\" (UniqueName: \"kubernetes.io/projected/fcc1787d-a277-48fe-b6ea-8dde75b794ef-kube-api-access-znr5q\") pod \"node-ca-jqvjm\" (UID: \"fcc1787d-a277-48fe-b6ea-8dde75b794ef\") " pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.293849 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.293715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lc58\" (UniqueName: \"kubernetes.io/projected/7b0bfc28-5d5c-42b3-aada-e8f37187da96-kube-api-access-7lc58\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:17.294946 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.294922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnlzx\" (UniqueName: \"kubernetes.io/projected/6c45d7ef-4d75-46f9-8874-2e01a2a5945b-kube-api-access-jnlzx\") pod \"iptables-alerter-8rgnm\" (UID: \"6c45d7ef-4d75-46f9-8874-2e01a2a5945b\") " pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.295417 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.295371    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtc8f\" (UniqueName: \"kubernetes.io/projected/2a5f5bf2-ff2e-49c2-b688-b1b298de105d-kube-api-access-mtc8f\") pod \"multus-additional-cni-plugins-2dpm8\" (UID: \"2a5f5bf2-ff2e-49c2-b688-b1b298de105d\") " pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.459597 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.459508    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8rgnm"
Jul 10 18:39:17.465676 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.465645    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xbw6g"
Jul 10 18:39:17.475361 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.475345    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xh9z9"
Jul 10 18:39:17.480128 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.480106    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jqvjm"
Jul 10 18:39:17.486633 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.486613    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2dpm8"
Jul 10 18:39:17.493327 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.493309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:17.500067 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.500050    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7"
Jul 10 18:39:17.506001 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.505981    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5kd9"
Jul 10 18:39:17.511516 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.511498    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:17.765673 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:17.765633    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd859782_385f_4f5b_96cd_f10e405bfdbb.slice/crio-c352f2e35d96b1e1d35a9e29ff6fb9432429f9fcdbe355ad8fbc1c7a42ab437f WatchSource:0}: Error finding container c352f2e35d96b1e1d35a9e29ff6fb9432429f9fcdbe355ad8fbc1c7a42ab437f: Status 404 returned error can't find the container with id c352f2e35d96b1e1d35a9e29ff6fb9432429f9fcdbe355ad8fbc1c7a42ab437f
Jul 10 18:39:17.769812 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:17.769788    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae3b187_d48f_43e9_a6a3_8bd2dae78715.slice/crio-06606c151322df5918c1cc753dce1b99bad15fe0b3bb849758244322fdd488c3 WatchSource:0}: Error finding container 06606c151322df5918c1cc753dce1b99bad15fe0b3bb849758244322fdd488c3: Status 404 returned error can't find the container with id 06606c151322df5918c1cc753dce1b99bad15fe0b3bb849758244322fdd488c3
Jul 10 18:39:17.770552 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:17.770530    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ec85bc_771c_42e6_8c4b_6a31203c35c9.slice/crio-b9bc4e598913d2f6498c4f5b436e8dd7e7e77ae672c2ce5dcbd1ce5df771e749 WatchSource:0}: Error finding container b9bc4e598913d2f6498c4f5b436e8dd7e7e77ae672c2ce5dcbd1ce5df771e749: Status 404 returned error can't find the container with id b9bc4e598913d2f6498c4f5b436e8dd7e7e77ae672c2ce5dcbd1ce5df771e749
Jul 10 18:39:17.771231 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:17.771158    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc1787d_a277_48fe_b6ea_8dde75b794ef.slice/crio-77bd34f491be1b3f1b6b56836af4314f3cc1a67546000e2c56d0e2e77e6d71d2 WatchSource:0}: Error finding container 77bd34f491be1b3f1b6b56836af4314f3cc1a67546000e2c56d0e2e77e6d71d2: Status 404 returned error can't find the container with id 77bd34f491be1b3f1b6b56836af4314f3cc1a67546000e2c56d0e2e77e6d71d2
Jul 10 18:39:17.780289 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.780228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:17.780289 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.780267    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:17.780414 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.780365    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:17.780414 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.780404    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:17.780557 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.780426    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret podName:2d494c4a-3c15-495c-a993-9bbf592b6d50 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:18.780405729 +0000 UTC m=+4.042070493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret") pod "global-pull-secret-syncer-bcvk5" (UID: "2d494c4a-3c15-495c-a993-9bbf592b6d50") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:17.780557 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.780450    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:18.780433816 +0000 UTC m=+4.042098567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:17.781367 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:17.781341    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6b9b9c_4522_417a_9f8c_89a5cdc7cd7f.slice/crio-4a83e8286210f58664fdf6afa23540d2f92098ef27597f80e6b00a35b7304be8 WatchSource:0}: Error finding container 4a83e8286210f58664fdf6afa23540d2f92098ef27597f80e6b00a35b7304be8: Status 404 returned error can't find the container with id 4a83e8286210f58664fdf6afa23540d2f92098ef27597f80e6b00a35b7304be8
Jul 10 18:39:17.782507 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:17.782474    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88cc0f38_18da_4e79_b96f_5b6435cdf0d7.slice/crio-fd386a3ea7433f7b953a4a9f2f6f206fcef3099318b9e0f3b0afd125090bb1cd WatchSource:0}: Error finding container fd386a3ea7433f7b953a4a9f2f6f206fcef3099318b9e0f3b0afd125090bb1cd: Status 404 returned error can't find the container with id fd386a3ea7433f7b953a4a9f2f6f206fcef3099318b9e0f3b0afd125090bb1cd
Jul 10 18:39:17.881095 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:17.881067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:17.881267 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.881247    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:17.881347 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.881274    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:17.881347 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.881287    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9njsl for pod openshift-network-diagnostics/network-check-target-p7nlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:17.881442 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:17.881351    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl podName:5a7bf1fa-bfba-4190-be81-f7bada15234e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:18.881331239 +0000 UTC m=+4.142996004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9njsl" (UniqueName: "kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl") pod "network-check-target-p7nlk" (UID: "5a7bf1fa-bfba-4190-be81-f7bada15234e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:18.198334 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.198276    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:34:17 +0000 UTC" deadline="2027-12-05 16:06:13.605947518 +0000 UTC"
Jul 10 18:39:18.198334 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.198332    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12309h26m55.407619386s"
Jul 10 18:39:18.267766 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.267453    2558 generic.go:356] "Generic (PLEG): container finished" podID="8b171dc9bc69a4139a3473bd963deb50" containerID="7a3337a96bcc22c2f3e1acb594160f43a142cd903e7534ce7ebd9d2ca8d016d6" exitCode=0
Jul 10 18:39:18.267766 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.267550    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" event={"ID":"8b171dc9bc69a4139a3473bd963deb50","Type":"ContainerDied","Data":"7a3337a96bcc22c2f3e1acb594160f43a142cd903e7534ce7ebd9d2ca8d016d6"}
Jul 10 18:39:18.270817 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.270566    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7" event={"ID":"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f","Type":"ContainerStarted","Data":"4a83e8286210f58664fdf6afa23540d2f92098ef27597f80e6b00a35b7304be8"}
Jul 10 18:39:18.273190 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.273129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbw6g" event={"ID":"ddf22656-c128-49dc-80fb-94a983d5ebad","Type":"ContainerStarted","Data":"e7d01f37acf962ddde7fe88c93cbeb7e98d530b0d694e383e20d8f910db0e8a0"}
Jul 10 18:39:18.276220 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.276188    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jqvjm" event={"ID":"fcc1787d-a277-48fe-b6ea-8dde75b794ef","Type":"ContainerStarted","Data":"77bd34f491be1b3f1b6b56836af4314f3cc1a67546000e2c56d0e2e77e6d71d2"}
Jul 10 18:39:18.278391 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.278363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"b9bc4e598913d2f6498c4f5b436e8dd7e7e77ae672c2ce5dcbd1ce5df771e749"}
Jul 10 18:39:18.280355 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.280331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sqq97" event={"ID":"fae3b187-d48f-43e9-a6a3-8bd2dae78715","Type":"ContainerStarted","Data":"06606c151322df5918c1cc753dce1b99bad15fe0b3bb849758244322fdd488c3"}
Jul 10 18:39:18.284166 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.284140    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5kd9" event={"ID":"88cc0f38-18da-4e79-b96f-5b6435cdf0d7","Type":"ContainerStarted","Data":"fd386a3ea7433f7b953a4a9f2f6f206fcef3099318b9e0f3b0afd125090bb1cd"}
Jul 10 18:39:18.288534 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.288500    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerStarted","Data":"9318595e8f022a2899884541435b4e67b6fb94119e9f20281713e7461f1cfc6c"}
Jul 10 18:39:18.289930 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.289903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8rgnm" event={"ID":"6c45d7ef-4d75-46f9-8874-2e01a2a5945b","Type":"ContainerStarted","Data":"57a0741ab144cb3e421aff61a871301c2d7ce67dd64100b7f4ba49ee94f789fe"}
Jul 10 18:39:18.292090 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.292064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xh9z9" event={"ID":"cd859782-385f-4f5b-96cd-f10e405bfdbb","Type":"ContainerStarted","Data":"c352f2e35d96b1e1d35a9e29ff6fb9432429f9fcdbe355ad8fbc1c7a42ab437f"}
Jul 10 18:39:18.787841 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.787798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:18.788020 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.787851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:18.788020 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.787979    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:18.788134 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.788038    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:20.788020542 +0000 UTC m=+6.049685299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:18.788134 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.788104    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:18.788243 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.788139    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret podName:2d494c4a-3c15-495c-a993-9bbf592b6d50 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:20.788128172 +0000 UTC m=+6.049792938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret") pod "global-pull-secret-syncer-bcvk5" (UID: "2d494c4a-3c15-495c-a993-9bbf592b6d50") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:18.889409 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:18.888846    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:18.889409 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.889005    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:18.889409 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.889026    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:18.889409 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.889038    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9njsl for pod openshift-network-diagnostics/network-check-target-p7nlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:18.889409 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:18.889091    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl podName:5a7bf1fa-bfba-4190-be81-f7bada15234e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:20.889074484 +0000 UTC m=+6.150739249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9njsl" (UniqueName: "kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl") pod "network-check-target-p7nlk" (UID: "5a7bf1fa-bfba-4190-be81-f7bada15234e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:19.198986 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:19.198876    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:34:17 +0000 UTC" deadline="2028-04-03 19:10:40.31158854 +0000 UTC"
Jul 10 18:39:19.198986 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:19.198912    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15192h31m21.112680695s"
Jul 10 18:39:19.254442 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:19.254178    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:19.254442 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:19.254297    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:19.266776 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:19.257865    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:19.266776 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:19.258043    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:19.266776 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:19.258116    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:19.266776 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:19.258218    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:19.327859 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:19.327824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" event={"ID":"8b171dc9bc69a4139a3473bd963deb50","Type":"ContainerStarted","Data":"475b43ebbe25fb83479071827c379b57734bc40c82ada7c2276fc9351d3301d6"}
Jul 10 18:39:19.343722 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:19.343660    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.ec2.internal" podStartSLOduration=3.343641361 podStartE2EDuration="3.343641361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:19.342906516 +0000 UTC m=+4.604571290" watchObservedRunningTime="2026-07-10 18:39:19.343641361 +0000 UTC m=+4.605306135"
Jul 10 18:39:20.804678 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:20.804639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:20.805149 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:20.804692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:20.805149 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.804866    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:20.805149 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.804928    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:24.804908439 +0000 UTC m=+10.066573194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:20.805338 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.805311    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:20.805397 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.805364    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret podName:2d494c4a-3c15-495c-a993-9bbf592b6d50 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:24.805349904 +0000 UTC m=+10.067014662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret") pod "global-pull-secret-syncer-bcvk5" (UID: "2d494c4a-3c15-495c-a993-9bbf592b6d50") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:20.905815 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:20.905774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:20.905973 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.905935    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:20.905973 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.905953    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:20.905973 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.905967    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9njsl for pod openshift-network-diagnostics/network-check-target-p7nlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:20.906143 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:20.906019    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl podName:5a7bf1fa-bfba-4190-be81-f7bada15234e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:24.906002415 +0000 UTC m=+10.167667183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9njsl" (UniqueName: "kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl") pod "network-check-target-p7nlk" (UID: "5a7bf1fa-bfba-4190-be81-f7bada15234e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:21.252802 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:21.252715    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:21.252802 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:21.252751    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:21.252994 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:21.252718    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:21.252994 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:21.252861    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:21.252994 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:21.252945    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:21.253149 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:21.253033    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:23.252793 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:23.252761    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:23.252793 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:23.252779    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:23.253315 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:23.252761    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:23.253315 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:23.252889    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:23.253315 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:23.252963    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:23.253315 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:23.253072    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:24.837594 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:24.837533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:24.838013 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:24.837599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:24.838013 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.837684    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:24.838013 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.837773    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret podName:2d494c4a-3c15-495c-a993-9bbf592b6d50 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:32.837752836 +0000 UTC m=+18.099417605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret") pod "global-pull-secret-syncer-bcvk5" (UID: "2d494c4a-3c15-495c-a993-9bbf592b6d50") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:24.838013 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.837788    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:24.838013 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.837853    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:32.837835654 +0000 UTC m=+18.099500419 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:24.938063 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:24.938022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:24.938211 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.938190    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:24.938403 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.938215    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:24.938403 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.938230    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9njsl for pod openshift-network-diagnostics/network-check-target-p7nlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:24.938403 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:24.938285    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl podName:5a7bf1fa-bfba-4190-be81-f7bada15234e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:32.938268624 +0000 UTC m=+18.199933398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9njsl" (UniqueName: "kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl") pod "network-check-target-p7nlk" (UID: "5a7bf1fa-bfba-4190-be81-f7bada15234e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:25.251864 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:25.251786    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:25.252007 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:25.251907    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:25.253438 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:25.253405    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:25.253567 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:25.253530    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:25.253633 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:25.253596    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:25.253684 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:25.253666    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:27.257419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:27.257386    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:27.257902 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:27.257569    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:27.258732 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:27.258029    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:27.258732 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:27.258148    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:27.258732 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:27.258213    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:27.258732 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:27.258300    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:29.251845 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:29.251813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:29.251845 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:29.251829    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:29.252266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:29.251821    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:29.252266 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:29.251928    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:29.252266 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:29.251998    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:29.252266 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:29.252066    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:31.252806 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:31.252771    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:31.252806 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:31.252779    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:31.253596 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:31.252830    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:31.253596 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:31.252925    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:31.253596 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:31.253017    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:31.253596 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:31.253124    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:32.902705 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:32.902669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:32.902705 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:32.902705    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:32.903188 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:32.902848    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:32.903188 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:32.902906    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:48.902890016 +0000 UTC m=+34.164554785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:32.903188 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:32.902849    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:32.903188 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:32.902937    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret podName:2d494c4a-3c15-495c-a993-9bbf592b6d50 nodeName:}" failed. No retries permitted until 2026-07-10 18:39:48.902931719 +0000 UTC m=+34.164596484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret") pod "global-pull-secret-syncer-bcvk5" (UID: "2d494c4a-3c15-495c-a993-9bbf592b6d50") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:33.003658 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:33.003618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:33.003840 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:33.003821    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:33.003904 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:33.003848    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:33.003904 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:33.003861    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9njsl for pod openshift-network-diagnostics/network-check-target-p7nlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:33.004008 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:33.003923    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl podName:5a7bf1fa-bfba-4190-be81-f7bada15234e nodeName:}" failed. No retries permitted until 2026-07-10 18:39:49.003903977 +0000 UTC m=+34.265568955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9njsl" (UniqueName: "kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl") pod "network-check-target-p7nlk" (UID: "5a7bf1fa-bfba-4190-be81-f7bada15234e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:33.252565 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:33.252486    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:33.252714 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:33.252488    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:33.252714 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:33.252618    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:33.252714 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:33.252617    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:33.252714 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:33.252705    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:33.252927 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:33.252795    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:35.253276 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:35.253245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:35.253731 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:35.253363    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:35.253731 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:35.253451    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:35.253731 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:35.253550    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:35.253731 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:35.253582    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:35.253731 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:35.253630    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:37.252079 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.252031    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:37.252380 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.252040    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:37.252380 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:37.252133    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:37.252380 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.252174    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:37.252380 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:37.252265    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:37.252380 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:37.252340    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:37.385371 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.385184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xh9z9" event={"ID":"cd859782-385f-4f5b-96cd-f10e405bfdbb","Type":"ContainerStarted","Data":"78c90195d78fba006dbe07a755046dc4867f3af67d0e2ecf677cb57a660cf9fa"}
Jul 10 18:39:37.387666 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.387510    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7" event={"ID":"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f","Type":"ContainerStarted","Data":"9a801768ccd04a38296bd4d158600a5b6bd874b371f92b25f29811feadd27129"}
Jul 10 18:39:37.389572 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.389549    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xbw6g" event={"ID":"ddf22656-c128-49dc-80fb-94a983d5ebad","Type":"ContainerStarted","Data":"7d16ed6ea891a5c34db71451016151e42c8ccd30b57daf9ce07acd480d1c4d79"}
Jul 10 18:39:37.391466 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.391442    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jqvjm" event={"ID":"fcc1787d-a277-48fe-b6ea-8dde75b794ef","Type":"ContainerStarted","Data":"922c9a36ff06d53c69dbc94ae0f0776d1c1bd09fe3902a89911b3eff94a21686"}
Jul 10 18:39:37.393167 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.393136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sqq97" event={"ID":"fae3b187-d48f-43e9-a6a3-8bd2dae78715","Type":"ContainerStarted","Data":"614a767342f4c18d457eb46627c2622f55696ee1a57cd9904ea668731eb065e1"}
Jul 10 18:39:37.394583 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.394554    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" event={"ID":"4084b00d8aece702c3f7f6e211f96825","Type":"ContainerStarted","Data":"8f436335a904290d3294344fe680185a942868509412a9c1a0e2e6e5358ee2f8"}
Jul 10 18:39:37.395837 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.395815    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5kd9" event={"ID":"88cc0f38-18da-4e79-b96f-5b6435cdf0d7","Type":"ContainerStarted","Data":"507be1a6dac0fd764357eac64d4cab646bcacc27ccc31b6e5c7f96fdfa2ce404"}
Jul 10 18:39:37.397142 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.397122    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerStarted","Data":"7cad6aaaa2ea7a6dbe4b40082592a1b50c2cb9f856fea7b65dc5bd1a0ccf8ff3"}
Jul 10 18:39:37.416069 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.416031    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xh9z9" podStartSLOduration=3.174051456 podStartE2EDuration="22.416018678s" totalImagesPullingTime="19.241967222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:37.415659408 +0000 UTC m=+22.677324182" watchObservedRunningTime="2026-07-10 18:39:37.416018678 +0000 UTC m=+22.677683450"
Jul 10 18:39:37.430185 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.430140    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.ec2.internal" podStartSLOduration=21.430126454 podStartE2EDuration="21.430126454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:37.42968777 +0000 UTC m=+22.691352543" watchObservedRunningTime="2026-07-10 18:39:37.430126454 +0000 UTC m=+22.691791227"
Jul 10 18:39:37.473530 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.473486    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sqq97" podStartSLOduration=3.244674242 podStartE2EDuration="22.473469635s" totalImagesPullingTime="19.228795393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:37.473391745 +0000 UTC m=+22.735056518" watchObservedRunningTime="2026-07-10 18:39:37.473469635 +0000 UTC m=+22.735134407"
Jul 10 18:39:37.491321 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.491276    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xbw6g" podStartSLOduration=3.25925949 podStartE2EDuration="22.491262691s" totalImagesPullingTime="19.232003201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:37.490953973 +0000 UTC m=+22.752618747" watchObservedRunningTime="2026-07-10 18:39:37.491262691 +0000 UTC m=+22.752927464"
Jul 10 18:39:37.505548 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.505510    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jqvjm" podStartSLOduration=3.273308954 podStartE2EDuration="22.505496162s" totalImagesPullingTime="19.232187208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:37.505279724 +0000 UTC m=+22.766944498" watchObservedRunningTime="2026-07-10 18:39:37.505496162 +0000 UTC m=+22.767160935"
Jul 10 18:39:37.521963 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:37.521916    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g5kd9" podStartSLOduration=3.298701003 podStartE2EDuration="22.521903578s" totalImagesPullingTime="19.223202575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:37.521494555 +0000 UTC m=+22.783159331" watchObservedRunningTime="2026-07-10 18:39:37.521903578 +0000 UTC m=+22.783568351"
Jul 10 18:39:38.400402 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.400232    2558 generic.go:356] "Generic (PLEG): container finished" podID="2a5f5bf2-ff2e-49c2-b688-b1b298de105d" containerID="7cad6aaaa2ea7a6dbe4b40082592a1b50c2cb9f856fea7b65dc5bd1a0ccf8ff3" exitCode=0
Jul 10 18:39:38.401092 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.400319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerDied","Data":"7cad6aaaa2ea7a6dbe4b40082592a1b50c2cb9f856fea7b65dc5bd1a0ccf8ff3"}
Jul 10 18:39:38.402103 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.402078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8rgnm" event={"ID":"6c45d7ef-4d75-46f9-8874-2e01a2a5945b","Type":"ContainerStarted","Data":"c36d5a68b210cd40fb81c514fb68f43f83ce89c614d7a38086d29748c5fe0c54"}
Jul 10 18:39:38.404825 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.404808    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:39:38.405165 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.405138    2558 generic.go:356] "Generic (PLEG): container finished" podID="86ec85bc-771c-42e6-8c4b-6a31203c35c9" containerID="523cb5c41aed9586fb674d721a49065b34b771945f1d16ae0073e1dbc8770357" exitCode=1
Jul 10 18:39:38.405261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.405236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"ab9e9e1d4b2b8083f20d29803601356cbceefe373754ae5a50544aa04e9c54c8"}
Jul 10 18:39:38.405320 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.405290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"fed9a279fd3954ce39df1a3865700429e7a06441f117da7c9ee782c60aca0e01"}
Jul 10 18:39:38.405371 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.405321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"71b064cbf25d0d70652141e99d4a9ce5e1ccf198fe5c7deff67b36a10bd957d7"}
Jul 10 18:39:38.405371 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.405350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"8b9014fc775cd58fb9c2be04c29408fe6292e3fadc6b93a6d04e15f214eb0e55"}
Jul 10 18:39:38.405447 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.405378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerDied","Data":"523cb5c41aed9586fb674d721a49065b34b771945f1d16ae0073e1dbc8770357"}
Jul 10 18:39:38.405447 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.405410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"64cf18d39568ddf9a79c56186af75dcc4a373c6ca79d4e129ca400b845780c55"}
Jul 10 18:39:38.439761 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.439702    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8rgnm" podStartSLOduration=4.208621986 podStartE2EDuration="23.43968933s" totalImagesPullingTime="19.231067344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:38.439430039 +0000 UTC m=+23.701094824" watchObservedRunningTime="2026-07-10 18:39:38.43968933 +0000 UTC m=+23.701354103"
Jul 10 18:39:38.606969 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:38.606940    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 18:39:39.252177 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:39.252072    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T18:39:38.606963869Z","UUID":"b1844dde-f27d-4c99-8c50-c489924c5ce9","Handler":null,"Name":"","Endpoint":""}
Jul 10 18:39:39.252567 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:39.252335    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:39.252567 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:39.252445    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:39.252567 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:39.252514    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:39.252567 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:39.252537    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:39.252799 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:39.252591    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:39.252799 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:39.252671    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:39.254320 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:39.254297    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 18:39:39.254445 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:39.254342    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 18:39:39.410919 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:39.410882    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7" event={"ID":"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f","Type":"ContainerStarted","Data":"4a0252e19ab4d78561014c703647243f0e77e6d90e6c4d6861f290724be23009"}
Jul 10 18:39:40.317683 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:40.317524    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:40.318115 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:40.318098    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:40.415712 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:40.415640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7" event={"ID":"db6b9b9c-4522-417a-9f8c-89a5cdc7cd7f","Type":"ContainerStarted","Data":"c5ebe86c477b0e1458b88c0967ea6b3eb060d8d0f293b5d4d5d4672e95a10504"}
Jul 10 18:39:40.419015 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:40.418991    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:39:40.419444 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:40.419422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"0c9b38de21b3e51d8d5a8f2f14affff41ead0a6016dbfddb210b48111d8ec015"}
Jul 10 18:39:40.435648 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:40.435605    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jk5b7" podStartSLOduration=3.479863495 podStartE2EDuration="25.435591278s" totalImagesPullingTime="21.955727783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:40.434990681 +0000 UTC m=+25.696655453" watchObservedRunningTime="2026-07-10 18:39:40.435591278 +0000 UTC m=+25.697256051"
Jul 10 18:39:41.252120 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:41.252085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:41.252434 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:41.252092    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:41.252434 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:41.252205    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:41.252434 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:41.252097    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:41.252434 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:41.252267    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:41.252434 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:41.252368    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:43.252838 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.252809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:43.253591 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.252814    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:43.253591 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.252927    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:43.253591 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:43.252920    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:43.253591 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:43.253020    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:43.253591 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:43.253095    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:43.429754 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.429726    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:39:43.430142 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.430113    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"df60c97efd647cd55c2d9c05661c0f1c14413a45cbc8ac4914d1de41bb36b542"}
Jul 10 18:39:43.430421 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.430404    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:43.430632 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.430613    2558 scope.go:117] "RemoveContainer" containerID="523cb5c41aed9586fb674d721a49065b34b771945f1d16ae0073e1dbc8770357"
Jul 10 18:39:43.449026 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.449009    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:43.679913 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.679885    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:43.680011 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.680001    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 10 18:39:43.680491 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:43.680470    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sqq97"
Jul 10 18:39:44.437182 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.437098    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:39:44.437702 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.437536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" event={"ID":"86ec85bc-771c-42e6-8c4b-6a31203c35c9","Type":"ContainerStarted","Data":"2f17c1308f511e37dfe59e05e0d4eafa117ad32bebeb91045b88823230d4ddfb"}
Jul 10 18:39:44.437819 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.437805    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:44.437877 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.437864    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:44.439657 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.439628    2558 generic.go:356] "Generic (PLEG): container finished" podID="2a5f5bf2-ff2e-49c2-b688-b1b298de105d" containerID="e99e736a59b603b074da02d6a9b35bd7ea15c9090e3b33ed6ebb2c1b0e587887" exitCode=0
Jul 10 18:39:44.439793 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.439727    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerDied","Data":"e99e736a59b603b074da02d6a9b35bd7ea15c9090e3b33ed6ebb2c1b0e587887"}
Jul 10 18:39:44.470670 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.470610    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt" podStartSLOduration=9.858982591 podStartE2EDuration="29.470589113s" totalImagesPullingTime="19.611606522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:44.467292069 +0000 UTC m=+29.728956866" watchObservedRunningTime="2026-07-10 18:39:44.470589113 +0000 UTC m=+29.732253886"
Jul 10 18:39:44.472100 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:44.472082    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:39:45.253395 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.252858    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:45.253395 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:45.252988    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:45.253640 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.253450    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:45.253640 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:45.253556    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:45.253640 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.253622    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:45.253801 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:45.253781    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:45.265522 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.265495    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bcvk5"]
Jul 10 18:39:45.268145 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.268120    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p7nlk"]
Jul 10 18:39:45.271220 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.271197    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j7qn6"]
Jul 10 18:39:45.442138 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.442114    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:45.442138 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.442138    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:45.442553 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:45.442234    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:45.442553 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:45.442288    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:45.442553 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:45.442311    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:45.442553 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:45.442382    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:46.446445 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:46.446412    2558 generic.go:356] "Generic (PLEG): container finished" podID="2a5f5bf2-ff2e-49c2-b688-b1b298de105d" containerID="cffa214c3d992543fa79c2d22ecb260474d3a0a795cde4eb84e50b2428a8e2fa" exitCode=0
Jul 10 18:39:46.446780 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:46.446492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerDied","Data":"cffa214c3d992543fa79c2d22ecb260474d3a0a795cde4eb84e50b2428a8e2fa"}
Jul 10 18:39:47.252754 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:47.252723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:47.252922 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:47.252782    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:47.252922 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:47.252857    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:47.253031 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:47.252930    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:47.253031 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:47.252989    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:47.253132 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:47.253067    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:48.452412 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:48.452384    2558 generic.go:356] "Generic (PLEG): container finished" podID="2a5f5bf2-ff2e-49c2-b688-b1b298de105d" containerID="700c35720a6bf00ad4a2a6a425ef02c31ef7a73b885e156fd423f5de7b9a038f" exitCode=0
Jul 10 18:39:48.452938 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:48.452431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerDied","Data":"700c35720a6bf00ad4a2a6a425ef02c31ef7a73b885e156fd423f5de7b9a038f"}
Jul 10 18:39:48.933526 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:48.933446    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:48.933526 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:48.933490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:48.933730 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:48.933599    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:48.933730 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:48.933621    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:48.933730 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:48.933659    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:20.933638538 +0000 UTC m=+66.195303292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 18:39:48.933730 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:48.933686    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret podName:2d494c4a-3c15-495c-a993-9bbf592b6d50 nodeName:}" failed. No retries permitted until 2026-07-10 18:40:20.933666587 +0000 UTC m=+66.195331355 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret") pod "global-pull-secret-syncer-bcvk5" (UID: "2d494c4a-3c15-495c-a993-9bbf592b6d50") : object "kube-system"/"original-pull-secret" not registered
Jul 10 18:39:49.033954 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:49.033920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:49.034125 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:49.034042    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 18:39:49.034125 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:49.034068    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 18:39:49.034125 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:49.034081    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9njsl for pod openshift-network-diagnostics/network-check-target-p7nlk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:49.034285 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:49.034142    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl podName:5a7bf1fa-bfba-4190-be81-f7bada15234e nodeName:}" failed. No retries permitted until 2026-07-10 18:40:21.034123181 +0000 UTC m=+66.295787945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9njsl" (UniqueName: "kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl") pod "network-check-target-p7nlk" (UID: "5a7bf1fa-bfba-4190-be81-f7bada15234e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 18:39:49.252897 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:49.252816    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:49.252897 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:49.252860    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:49.253117 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:49.252816    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:49.253117 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:49.252944    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p7nlk" podUID="5a7bf1fa-bfba-4190-be81-f7bada15234e"
Jul 10 18:39:49.253117 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:49.253015    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j7qn6" podUID="7b0bfc28-5d5c-42b3-aada-e8f37187da96"
Jul 10 18:39:49.253117 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:49.253104    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bcvk5" podUID="2d494c4a-3c15-495c-a993-9bbf592b6d50"
Jul 10 18:39:51.051562 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.051484    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-163.ec2.internal" event="NodeReady"
Jul 10 18:39:51.052165 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.051608    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 18:39:51.087970 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.087941    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f56675b7f-v8np5"]
Jul 10 18:39:51.090414 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.089958    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.091120 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.090958    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"]
Jul 10 18:39:51.093035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.092660    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 10 18:39:51.093035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.092822    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 10 18:39:51.093035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.092908    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tn8fv\""
Jul 10 18:39:51.093035 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.092938    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 10 18:39:51.093303 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.093197    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"]
Jul 10 18:39:51.093427 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.093393    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.094975 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.094950    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"]
Jul 10 18:39:51.095106 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.095084    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.097847 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.097388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 10 18:39:51.097847 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.097678    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 10 18:39:51.097847 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.097722    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 10 18:39:51.098094 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.097940    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 18:39:51.098094 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.097944    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 18:39:51.098221 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.098116    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 18:39:51.098297 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.098278    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 10 18:39:51.098797 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.098624    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 10 18:39:51.099614 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.099148    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-t9z88\""
Jul 10 18:39:51.100775 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.100736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.101167 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.101148    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 10 18:39:51.102957 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.102940    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 10 18:39:51.105853 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.105833    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"]
Jul 10 18:39:51.106731 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.106697    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f56675b7f-v8np5"]
Jul 10 18:39:51.107909 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.107855    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"]
Jul 10 18:39:51.112407 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.112386    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xr5nv"]
Jul 10 18:39:51.114452 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.114432    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"]
Jul 10 18:39:51.114606 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.114466    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hhxbq"]
Jul 10 18:39:51.114798 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.114785    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.116350 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.116334    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.117125 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.117106    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 18:39:51.117125 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.117117    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q5td4\""
Jul 10 18:39:51.117261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.117105    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 18:39:51.117261 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.117108    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 18:39:51.118191 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.118167    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 18:39:51.118433 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.118415    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 18:39:51.118538 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.118522    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s4tdx\""
Jul 10 18:39:51.123977 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.123953    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xr5nv"]
Jul 10 18:39:51.124651 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.124631    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hhxbq"]
Jul 10 18:39:51.250357 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250320    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.250506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.250506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250440    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-ca\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.250506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250473    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.250506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-trusted-ca\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.250694 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpzqj\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-kube-api-access-rpzqj\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.250694 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.250694 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rmf\" (UniqueName: \"kubernetes.io/projected/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-kube-api-access-q6rmf\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.250694 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6df8b103-18a2-427a-bb8e-815f73a5ebdb-ca-trust-extracted\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.250694 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzz69\" (UniqueName: \"kubernetes.io/projected/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-kube-api-access-dzz69\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250702    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-hub\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f070177-34ce-4586-94ed-96206e46da76-cert\") pod \"ingress-canary-xr5nv\" (UID: \"9f070177-34ce-4586-94ed-96206e46da76\") " pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250813    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e1cdc-4e50-4260-9838-df169c0b9ade-tmp-dir\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250837    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-installation-pull-secrets\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250893    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tnpv\" (UniqueName: \"kubernetes.io/projected/9f070177-34ce-4586-94ed-96206e46da76-kube-api-access-8tnpv\") pod \"ingress-canary-xr5nv\" (UID: \"9f070177-34ce-4586-94ed-96206e46da76\") " pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.250953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-bound-sa-token\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.251214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7e1cdc-4e50-4260-9838-df169c0b9ade-config-volume\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.251214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.250998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknjw\" (UniqueName: \"kubernetes.io/projected/2e00f5b9-e603-437b-bd53-45ee2950a580-kube-api-access-wknjw\") pod \"managed-serviceaccount-addon-agent-6b5d488456-5lg5z\" (UID: \"2e00f5b9-e603-437b-bd53-45ee2950a580\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.251214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.251032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqrg\" (UniqueName: \"kubernetes.io/projected/ef7e1cdc-4e50-4260-9838-df169c0b9ade-kube-api-access-fbqrg\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.251214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.251064    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-image-registry-private-configuration\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.251214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.251105    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e00f5b9-e603-437b-bd53-45ee2950a580-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5d488456-5lg5z\" (UID: \"2e00f5b9-e603-437b-bd53-45ee2950a580\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.251214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.251124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-certificates\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.251214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.251142    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-tmp\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.252488 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.252463    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:39:51.252488 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.252480    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:39:51.252681 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.252501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:39:51.255108 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.255068    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:39:51.255108 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.255105    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crdnw\""
Jul 10 18:39:51.255378 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.255355    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:39:51.255461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.255395    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-75lnr\""
Jul 10 18:39:51.255461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.255440    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:39:51.255597 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.255514    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:39:51.351736 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.351648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknjw\" (UniqueName: \"kubernetes.io/projected/2e00f5b9-e603-437b-bd53-45ee2950a580-kube-api-access-wknjw\") pod \"managed-serviceaccount-addon-agent-6b5d488456-5lg5z\" (UID: \"2e00f5b9-e603-437b-bd53-45ee2950a580\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.351736 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.351698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqrg\" (UniqueName: \"kubernetes.io/projected/ef7e1cdc-4e50-4260-9838-df169c0b9ade-kube-api-access-fbqrg\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.351974 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.351733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-image-registry-private-configuration\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.351974 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.351804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e00f5b9-e603-437b-bd53-45ee2950a580-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5d488456-5lg5z\" (UID: \"2e00f5b9-e603-437b-bd53-45ee2950a580\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.351974 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.351827    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-certificates\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.351974 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.351856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-tmp\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.351974 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.351889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.352247 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.352247 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352235    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-ca\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.352350 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352262    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.352350 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-trusted-ca\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.352350 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpzqj\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-kube-api-access-rpzqj\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.352506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.352506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rmf\" (UniqueName: \"kubernetes.io/projected/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-kube-api-access-q6rmf\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.352506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6df8b103-18a2-427a-bb8e-815f73a5ebdb-ca-trust-extracted\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.352506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz69\" (UniqueName: \"kubernetes.io/projected/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-kube-api-access-dzz69\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.352506 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352494    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-hub\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f070177-34ce-4586-94ed-96206e46da76-cert\") pod \"ingress-canary-xr5nv\" (UID: \"9f070177-34ce-4586-94ed-96206e46da76\") " pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352552    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e1cdc-4e50-4260-9838-df169c0b9ade-tmp-dir\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352598    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-installation-pull-secrets\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-certificates\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tnpv\" (UniqueName: \"kubernetes.io/projected/9f070177-34ce-4586-94ed-96206e46da76-kube-api-access-8tnpv\") pod \"ingress-canary-xr5nv\" (UID: \"9f070177-34ce-4586-94ed-96206e46da76\") " pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352678    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-bound-sa-token\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.352735 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.352721    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7e1cdc-4e50-4260-9838-df169c0b9ade-config-volume\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.354197 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.353254    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-trusted-ca\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.354197 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.353287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7e1cdc-4e50-4260-9838-df169c0b9ade-config-volume\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.354197 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.353349    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:39:51.354197 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.353403    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:39:51.853385933 +0000 UTC m=+37.115050704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:39:51.354197 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.353690    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:39:51.354197 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.353706    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:39:51.354197 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.353775    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:39:51.853735634 +0000 UTC m=+37.115400399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:39:51.355529 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.354718    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-tmp\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.355529 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.355027    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6df8b103-18a2-427a-bb8e-815f73a5ebdb-ca-trust-extracted\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.357184 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.357156    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-ca\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.357279 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.357223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-hub\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.357279 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.357246    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-image-registry-private-configuration\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.357387 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.357284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2e00f5b9-e603-437b-bd53-45ee2950a580-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6b5d488456-5lg5z\" (UID: \"2e00f5b9-e603-437b-bd53-45ee2950a580\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.357620 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.357577    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef7e1cdc-4e50-4260-9838-df169c0b9ade-tmp-dir\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.358283 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.357902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-additional-service-ca\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.358593 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.358554    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-klusterlet-config\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.358695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.358674    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f070177-34ce-4586-94ed-96206e46da76-cert\") pod \"ingress-canary-xr5nv\" (UID: \"9f070177-34ce-4586-94ed-96206e46da76\") " pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.359803 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.359780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-installation-pull-secrets\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.360101 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.360082    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.360578 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.360255    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.361087 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.361054    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqrg\" (UniqueName: \"kubernetes.io/projected/ef7e1cdc-4e50-4260-9838-df169c0b9ade-kube-api-access-fbqrg\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.361087 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.361061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknjw\" (UniqueName: \"kubernetes.io/projected/2e00f5b9-e603-437b-bd53-45ee2950a580-kube-api-access-wknjw\") pod \"managed-serviceaccount-addon-agent-6b5d488456-5lg5z\" (UID: \"2e00f5b9-e603-437b-bd53-45ee2950a580\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.363272 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.363013    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz69\" (UniqueName: \"kubernetes.io/projected/0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26-kube-api-access-dzz69\") pod \"cluster-proxy-proxy-agent-68d6f475bc-mmcz4\" (UID: \"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.363272 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.363233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rmf\" (UniqueName: \"kubernetes.io/projected/2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d-kube-api-access-q6rmf\") pod \"klusterlet-addon-workmgr-5bd6465b4c-pwtxp\" (UID: \"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.363559 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.363516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpzqj\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-kube-api-access-rpzqj\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.364461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.364437    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tnpv\" (UniqueName: \"kubernetes.io/projected/9f070177-34ce-4586-94ed-96206e46da76-kube-api-access-8tnpv\") pod \"ingress-canary-xr5nv\" (UID: \"9f070177-34ce-4586-94ed-96206e46da76\") " pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.365379 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.365355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-bound-sa-token\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.419504 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.419466    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:39:51.427777 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.427734    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"
Jul 10 18:39:51.439502 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.439478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:39:51.446170 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.446152    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xr5nv"
Jul 10 18:39:51.856584 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.856548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:51.856584 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:51.856589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:51.856857 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.856703    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:39:51.856857 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.856826    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:39:51.856857 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.856849    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:39:51.857010 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.856837    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:39:52.856819603 +0000 UTC m=+38.118484359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:39:51.857010 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:51.856943    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:39:52.856910372 +0000 UTC m=+38.118575131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:39:52.863245 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:52.862929    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:52.863245 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:52.863072    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:39:52.863245 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:52.863198    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:39:52.863608 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:52.863263    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:39:52.863608 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:52.863273    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:39:54.863251706 +0000 UTC m=+40.124916479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:39:52.863608 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:52.863176    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:52.863608 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:52.863320    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:39:54.863298076 +0000 UTC m=+40.124962831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:39:52.886497 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:52.886460    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z"]
Jul 10 18:39:52.889582 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:52.889557    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"]
Jul 10 18:39:52.893646 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:52.893617    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"]
Jul 10 18:39:52.900543 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:52.900494    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xr5nv"]
Jul 10 18:39:52.903816 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:52.903791    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cf52bbe_5f33_47b6_b0b3_49e6d8ab7c26.slice/crio-1d2d05d2b658dd231c922b68d395757114a546533f6a6e5adcac818ac7f7ea54 WatchSource:0}: Error finding container 1d2d05d2b658dd231c922b68d395757114a546533f6a6e5adcac818ac7f7ea54: Status 404 returned error can't find the container with id 1d2d05d2b658dd231c922b68d395757114a546533f6a6e5adcac818ac7f7ea54
Jul 10 18:39:52.904363 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:52.904343    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc058b7_fc3e_4d1a_ae8b_ee0adf225d3d.slice/crio-572006477f98a413fe4fd45c47622319eb57a8c57bbeefa731b4e3ed3b712cab WatchSource:0}: Error finding container 572006477f98a413fe4fd45c47622319eb57a8c57bbeefa731b4e3ed3b712cab: Status 404 returned error can't find the container with id 572006477f98a413fe4fd45c47622319eb57a8c57bbeefa731b4e3ed3b712cab
Jul 10 18:39:52.905328 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:52.905295    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f070177_34ce_4586_94ed_96206e46da76.slice/crio-85016826e16645617071afe461db2584b9d09f8f00abec1d1a39539e5c7b8cfe WatchSource:0}: Error finding container 85016826e16645617071afe461db2584b9d09f8f00abec1d1a39539e5c7b8cfe: Status 404 returned error can't find the container with id 85016826e16645617071afe461db2584b9d09f8f00abec1d1a39539e5c7b8cfe
Jul 10 18:39:52.905857 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:39:52.905834    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e00f5b9_e603_437b_bd53_45ee2950a580.slice/crio-63fdb956921b8ddc6b142e1dde4cb16ecb02c578a9bf8b810792c9c23105d4a5 WatchSource:0}: Error finding container 63fdb956921b8ddc6b142e1dde4cb16ecb02c578a9bf8b810792c9c23105d4a5: Status 404 returned error can't find the container with id 63fdb956921b8ddc6b142e1dde4cb16ecb02c578a9bf8b810792c9c23105d4a5
Jul 10 18:39:53.464914 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:53.464771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp" event={"ID":"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d","Type":"ContainerStarted","Data":"572006477f98a413fe4fd45c47622319eb57a8c57bbeefa731b4e3ed3b712cab"}
Jul 10 18:39:53.465848 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:53.465823    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xr5nv" event={"ID":"9f070177-34ce-4586-94ed-96206e46da76","Type":"ContainerStarted","Data":"85016826e16645617071afe461db2584b9d09f8f00abec1d1a39539e5c7b8cfe"}
Jul 10 18:39:53.468571 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:53.468547    2558 generic.go:356] "Generic (PLEG): container finished" podID="2a5f5bf2-ff2e-49c2-b688-b1b298de105d" containerID="2d09e1fe3fe369f6cd50ac9966e09710b8cb121f83ea74ff1f9ef937f2a31ef7" exitCode=0
Jul 10 18:39:53.468671 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:53.468618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerDied","Data":"2d09e1fe3fe369f6cd50ac9966e09710b8cb121f83ea74ff1f9ef937f2a31ef7"}
Jul 10 18:39:53.469792 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:53.469769    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" event={"ID":"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26","Type":"ContainerStarted","Data":"1d2d05d2b658dd231c922b68d395757114a546533f6a6e5adcac818ac7f7ea54"}
Jul 10 18:39:53.470822 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:53.470799    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z" event={"ID":"2e00f5b9-e603-437b-bd53-45ee2950a580","Type":"ContainerStarted","Data":"63fdb956921b8ddc6b142e1dde4cb16ecb02c578a9bf8b810792c9c23105d4a5"}
Jul 10 18:39:54.487275 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:54.486239    2558 generic.go:356] "Generic (PLEG): container finished" podID="2a5f5bf2-ff2e-49c2-b688-b1b298de105d" containerID="d15fc1ea688b81f6e6b3d48221056001b03c3386d8e09863dd31aebedb57846f" exitCode=0
Jul 10 18:39:54.487275 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:54.486308    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerDied","Data":"d15fc1ea688b81f6e6b3d48221056001b03c3386d8e09863dd31aebedb57846f"}
Jul 10 18:39:54.882113 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:54.881066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:54.882113 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:54.881318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:54.882113 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:54.881490    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:39:54.882113 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:54.881548    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:39:58.881530462 +0000 UTC m=+44.143195216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:39:54.882113 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:54.881975    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:39:54.882113 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:54.881993    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:39:54.882113 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:54.882040    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:39:58.882025006 +0000 UTC m=+44.143689760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:39:55.262942 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.262882    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqq97_fae3b187-d48f-43e9-a6a3-8bd2dae78715/konnectivity-agent/0.log" limit=1024
Jul 10 18:39:55.352644 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.352609    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqq97_fae3b187-d48f-43e9-a6a3-8bd2dae78715/konnectivity-agent/0.log" limit=1024
Jul 10 18:39:55.407285 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.407254    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqq97_fae3b187-d48f-43e9-a6a3-8bd2dae78715/konnectivity-agent/0.log" limit=1024
Jul 10 18:39:55.501781 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.501695    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" event={"ID":"2a5f5bf2-ff2e-49c2-b688-b1b298de105d","Type":"ContainerStarted","Data":"15e9bdf64607c69fcb31f0e223fc9c08f12b2cd3e71631a992ff556f1d5a3d84"}
Jul 10 18:39:55.527855 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.527601    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2dpm8" podStartSLOduration=9.955565396 podStartE2EDuration="40.5275802s" totalImagesPullingTime="30.572014804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:39:55.526224718 +0000 UTC m=+40.787889501" watchObservedRunningTime="2026-07-10 18:39:55.5275802 +0000 UTC m=+40.789244974"
Jul 10 18:39:55.651146 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.651049    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqq97_fae3b187-d48f-43e9-a6a3-8bd2dae78715/konnectivity-agent/0.log" limit=1024
Jul 10 18:39:55.709515 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.709363    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqq97_fae3b187-d48f-43e9-a6a3-8bd2dae78715/konnectivity-agent/0.log" limit=1024
Jul 10 18:39:55.916859 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:55.916776    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqq97_fae3b187-d48f-43e9-a6a3-8bd2dae78715/konnectivity-agent/0.log" limit=1024
Jul 10 18:39:58.910861 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:58.910824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:39:58.911396 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:39:58.910869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:39:58.911396 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:58.910975    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:39:58.911396 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:58.910986    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:39:58.911396 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:58.911004    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:39:58.911396 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:58.911033    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:40:06.911013999 +0000 UTC m=+52.172678750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:39:58.911396 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:39:58.911055    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:40:06.911046161 +0000 UTC m=+52.172710914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:40:01.519636 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.519599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z" event={"ID":"2e00f5b9-e603-437b-bd53-45ee2950a580","Type":"ContainerStarted","Data":"4414e38d7ae1ccc412650864afb929213d2be13edb96dd262af190746addebff"}
Jul 10 18:40:01.521112 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.521088    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp" event={"ID":"2fc058b7-fc3e-4d1a-ae8b-ee0adf225d3d","Type":"ContainerStarted","Data":"fe0a5e126a2bf964d12580f0e01eb5673338354f8d254142fe315e8e7b206d24"}
Jul 10 18:40:01.521231 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.521140    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:40:01.522661 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.522640    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xr5nv" event={"ID":"9f070177-34ce-4586-94ed-96206e46da76","Type":"ContainerStarted","Data":"f5db3626480e6c16c052e8b8af58d1c44f8f2d80bc3c819cea926545ec542f7c"}
Jul 10 18:40:01.523036 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.523018    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp"
Jul 10 18:40:01.524654 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.524634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" event={"ID":"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26","Type":"ContainerStarted","Data":"9d55ad82e228ac48650e84a766dd955f32a383c62b3a600adb46e0f04e5eb5d8"}
Jul 10 18:40:01.524730 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.524665    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" event={"ID":"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26","Type":"ContainerStarted","Data":"eb18a9f8662470e51c1630b749fae7d747bb3c27bcffc76d4dbfc61a4a5dccc0"}
Jul 10 18:40:01.524730 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.524678    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" event={"ID":"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26","Type":"ContainerStarted","Data":"150391fa61152f3f98612b2cd7127b4183c607caab0d53a2f6effc7d29274c32"}
Jul 10 18:40:01.557594 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.557515    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xr5nv" podStartSLOduration=2.432352202 podStartE2EDuration="10.557498999s" totalImagesPullingTime="8.125146797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:01.5569741 +0000 UTC m=+46.818638874" watchObservedRunningTime="2026-07-10 18:40:01.557498999 +0000 UTC m=+46.819163775"
Jul 10 18:40:01.557678 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.557624    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6b5d488456-5lg5z" podStartSLOduration=10.433501745 podStartE2EDuration="18.55761887s" totalImagesPullingTime="8.124117125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:01.541989415 +0000 UTC m=+46.803654190" watchObservedRunningTime="2026-07-10 18:40:01.55761887 +0000 UTC m=+46.819283701"
Jul 10 18:40:01.579682 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.578997    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" podStartSLOduration=10.452952148 podStartE2EDuration="18.578985082s" totalImagesPullingTime="8.126032934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:01.57628276 +0000 UTC m=+46.837947522" watchObservedRunningTime="2026-07-10 18:40:01.578985082 +0000 UTC m=+46.840649855"
Jul 10 18:40:01.594604 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:01.594552    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5bd6465b4c-pwtxp" podStartSLOduration=10.454761731 podStartE2EDuration="18.594539282s" totalImagesPullingTime="8.139777551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:01.59394792 +0000 UTC m=+46.855612693" watchObservedRunningTime="2026-07-10 18:40:01.594539282 +0000 UTC m=+46.856204032"
Jul 10 18:40:06.963633 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:06.963595    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:40:06.963633 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:06.963641    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:40:06.964128 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:06.963774    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:06.964128 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:06.963797    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:40:06.964128 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:06.963828    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:06.964128 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:06.963861    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:40:22.963846363 +0000 UTC m=+68.225511115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:40:06.964128 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:06.963875    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:40:22.963868996 +0000 UTC m=+68.225533746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:16.459510 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:16.459483    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4l7jt"
Jul 10 18:40:20.957910 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:20.957870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:40:20.957910 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:20.957911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:40:20.960541 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:20.960518    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 18:40:20.960541 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:20.960524    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 18:40:20.968631 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:20.968613    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 10 18:40:20.968711 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:20.968668    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs podName:7b0bfc28-5d5c-42b3-aada-e8f37187da96 nodeName:}" failed. No retries permitted until 2026-07-10 18:41:24.968653267 +0000 UTC m=+130.230318021 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs") pod "network-metrics-daemon-j7qn6" (UID: "7b0bfc28-5d5c-42b3-aada-e8f37187da96") : secret "metrics-daemon-secret" not found
Jul 10 18:40:20.972488 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:20.972469    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2d494c4a-3c15-495c-a993-9bbf592b6d50-original-pull-secret\") pod \"global-pull-secret-syncer-bcvk5\" (UID: \"2d494c4a-3c15-495c-a993-9bbf592b6d50\") " pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:40:21.058557 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.058520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:40:21.061052 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.061035    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 18:40:21.071367 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.071344    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 18:40:21.081316 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.081290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njsl\" (UniqueName: \"kubernetes.io/projected/5a7bf1fa-bfba-4190-be81-f7bada15234e-kube-api-access-9njsl\") pod \"network-check-target-p7nlk\" (UID: \"5a7bf1fa-bfba-4190-be81-f7bada15234e\") " pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:40:21.267092 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.267071    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crdnw\""
Jul 10 18:40:21.270157 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.270140    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bcvk5"
Jul 10 18:40:21.279830 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.275938    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:40:21.396629 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.396548    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bcvk5"]
Jul 10 18:40:21.399199 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:40:21.399173    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d494c4a_3c15_495c_a993_9bbf592b6d50.slice/crio-998f823b3c5b9bdb93eae8e611993f4c2a016a141ef676fa5a3fb1654f111e8c WatchSource:0}: Error finding container 998f823b3c5b9bdb93eae8e611993f4c2a016a141ef676fa5a3fb1654f111e8c: Status 404 returned error can't find the container with id 998f823b3c5b9bdb93eae8e611993f4c2a016a141ef676fa5a3fb1654f111e8c
Jul 10 18:40:21.414684 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.414627    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p7nlk"]
Jul 10 18:40:21.418095 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:40:21.418075    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7bf1fa_bfba_4190_be81_f7bada15234e.slice/crio-0d0245925d806d761d8bc5dd2d76468d13a9461758cb37f75664be1e27afcb0d WatchSource:0}: Error finding container 0d0245925d806d761d8bc5dd2d76468d13a9461758cb37f75664be1e27afcb0d: Status 404 returned error can't find the container with id 0d0245925d806d761d8bc5dd2d76468d13a9461758cb37f75664be1e27afcb0d
Jul 10 18:40:21.581991 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.581907    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p7nlk" event={"ID":"5a7bf1fa-bfba-4190-be81-f7bada15234e","Type":"ContainerStarted","Data":"0d0245925d806d761d8bc5dd2d76468d13a9461758cb37f75664be1e27afcb0d"}
Jul 10 18:40:21.582940 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:21.582910    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bcvk5" event={"ID":"2d494c4a-3c15-495c-a993-9bbf592b6d50","Type":"ContainerStarted","Data":"998f823b3c5b9bdb93eae8e611993f4c2a016a141ef676fa5a3fb1654f111e8c"}
Jul 10 18:40:22.973188 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:22.973142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:40:22.973649 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:22.973194    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:40:22.973649 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:22.973323    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:22.973649 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:22.973345    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:22.973649 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:22.973367    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:40:22.973649 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:22.973390    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:40:54.973371599 +0000 UTC m=+100.235036368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:22.973649 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:22.973443    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:40:54.973422991 +0000 UTC m=+100.235087745 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:40:25.598579 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:25.598522    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p7nlk" event={"ID":"5a7bf1fa-bfba-4190-be81-f7bada15234e","Type":"ContainerStarted","Data":"cebbdd97764c850065e28d0300ce54d14ba0c688807da4800fa4472519361573"}
Jul 10 18:40:25.599039 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:25.598669    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:40:25.615255 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:25.615203    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p7nlk" podStartSLOduration=67.476223425 podStartE2EDuration="1m10.615185366s" totalImagesPullingTime="3.138961941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:25.614445586 +0000 UTC m=+70.876110391" watchObservedRunningTime="2026-07-10 18:40:25.615185366 +0000 UTC m=+70.876850140"
Jul 10 18:40:27.605449 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:27.605362    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bcvk5" event={"ID":"2d494c4a-3c15-495c-a993-9bbf592b6d50","Type":"ContainerStarted","Data":"16915e31c227368cbda5e7a85e3b211f7876085deb64c0fb17702840a5dd42a3"}
Jul 10 18:40:27.619138 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:27.619071    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bcvk5" podStartSLOduration=66.747560963 podStartE2EDuration="1m12.619056812s" totalImagesPullingTime="5.871495849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:40:27.619023516 +0000 UTC m=+72.880688289" watchObservedRunningTime="2026-07-10 18:40:27.619056812 +0000 UTC m=+72.880721584"
Jul 10 18:40:54.991033 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:54.990994    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") pod \"image-registry-6f56675b7f-v8np5\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") " pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:40:54.991033 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:54.991032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:40:54.991460 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:54.991127    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 18:40:54.991460 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:54.991142    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 18:40:54.991460 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:54.991163    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f56675b7f-v8np5: secret "image-registry-tls" not found
Jul 10 18:40:54.991460 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:54.991178    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls podName:ef7e1cdc-4e50-4260-9838-df169c0b9ade nodeName:}" failed. No retries permitted until 2026-07-10 18:41:58.991164418 +0000 UTC m=+164.252829182 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls") pod "dns-default-hhxbq" (UID: "ef7e1cdc-4e50-4260-9838-df169c0b9ade") : secret "dns-default-metrics-tls" not found
Jul 10 18:40:54.991460 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:54.991206    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls podName:6df8b103-18a2-427a-bb8e-815f73a5ebdb nodeName:}" failed. No retries permitted until 2026-07-10 18:41:58.991192974 +0000 UTC m=+164.252857749 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls") pod "image-registry-6f56675b7f-v8np5" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb") : secret "image-registry-tls" not found
Jul 10 18:40:56.603640 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:56.603606    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p7nlk"
Jul 10 18:40:59.397580 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.397549    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rvxcl"]
Jul 10 18:40:59.423758 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.423722    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.427767 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.427725    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 18:40:59.427767 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.427762    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 18:40:59.428633 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.428615    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 18:40:59.428767 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.428728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m5ckx\""
Jul 10 18:40:59.435985 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.435964    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rvxcl"]
Jul 10 18:40:59.436367 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.436353    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 18:40:59.521617 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.521585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.521821 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.521639    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f1a2e54-5f36-425c-a557-8b78d8101f1e-data-volume\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.521821 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.521786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f1a2e54-5f36-425c-a557-8b78d8101f1e-crio-socket\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.521821 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.521820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.521963 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.521846    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmj8c\" (UniqueName: \"kubernetes.io/projected/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-api-access-pmj8c\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.622153 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.622116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f1a2e54-5f36-425c-a557-8b78d8101f1e-crio-socket\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.622153 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.622154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.622332 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.622180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmj8c\" (UniqueName: \"kubernetes.io/projected/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-api-access-pmj8c\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.622332 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.622205    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.622332 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.622246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f1a2e54-5f36-425c-a557-8b78d8101f1e-data-volume\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.622332 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:59.622255    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:40:59.622452 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:59.622336    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:59.622452 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:59.622341    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.122321041 +0000 UTC m=+105.383985796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : configmap "kube-rbac-proxy" not found
Jul 10 18:40:59.622452 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:40:59.622394    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:00.122383149 +0000 UTC m=+105.384047907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:40:59.622452 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.622254    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5f1a2e54-5f36-425c-a557-8b78d8101f1e-crio-socket\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.622585 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.622538    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f1a2e54-5f36-425c-a557-8b78d8101f1e-data-volume\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:40:59.648428 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:40:59.648367    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmj8c\" (UniqueName: \"kubernetes.io/projected/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-api-access-pmj8c\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:00.125690 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:00.125646    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:00.125943 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:00.125704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:00.125943 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:00.125848    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.125943 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:00.125877    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:00.125943 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:00.125931    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:01.125907575 +0000 UTC m=+106.387572329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:00.126112 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:00.125958    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:01.125938586 +0000 UTC m=+106.387603340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:01.131782 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:01.131733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:01.132255 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:01.131853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:01.132255 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:01.131880    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:01.132255 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:01.131943    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:01.132255 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:01.131950    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:03.131936106 +0000 UTC m=+108.393600862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:01.132255 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:01.131998    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:03.131978716 +0000 UTC m=+108.393643481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:01.391208 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:01.391136    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5kd9_88cc0f38-18da-4e79-b96f-5b6435cdf0d7/dns-node-resolver/0.log"
Jul 10 18:41:02.180114 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:02.180085    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jqvjm_fcc1787d-a277-48fe-b6ea-8dde75b794ef/node-ca/0.log"
Jul 10 18:41:03.148278 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:03.148244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:03.148467 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:03.148285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:03.148467 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:03.148372    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:03.148467 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:03.148437    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:03.148467 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:03.148445    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:07.148426629 +0000 UTC m=+112.410091381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:03.148678 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:03.148476    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:07.148465553 +0000 UTC m=+112.410130304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:03.381665 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:03.381635    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xr5nv_9f070177-34ce-4586-94ed-96206e46da76/serve-healthcheck-canary/0.log"
Jul 10 18:41:07.176252 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:07.176209    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:07.176709 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:07.176261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:07.176709 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:07.176322    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 18:41:07.176709 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:07.176385    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:15.176369591 +0000 UTC m=+120.438034342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : configmap "kube-rbac-proxy" not found
Jul 10 18:41:07.176709 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:07.176389    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:07.176709 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:07.176427    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:15.176416175 +0000 UTC m=+120.438080962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:15.236093 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:15.236056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:15.236635 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:15.236107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:15.236635 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:15.236212    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:15.236635 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:15.236287    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls podName:5f1a2e54-5f36-425c-a557-8b78d8101f1e nodeName:}" failed. No retries permitted until 2026-07-10 18:41:31.236269581 +0000 UTC m=+136.497934332 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rvxcl" (UID: "5f1a2e54-5f36-425c-a557-8b78d8101f1e") : secret "insights-runtime-extractor-tls" not found
Jul 10 18:41:15.236635 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:15.236497    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5f1a2e54-5f36-425c-a557-8b78d8101f1e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:21.421061 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:21.421009    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" podUID="0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 10 18:41:25.011091 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:25.011049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:41:25.013246 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:25.013222    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b0bfc28-5d5c-42b3-aada-e8f37187da96-metrics-certs\") pod \"network-metrics-daemon-j7qn6\" (UID: \"7b0bfc28-5d5c-42b3-aada-e8f37187da96\") " pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:41:25.178862 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:25.178831    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-75lnr\""
Jul 10 18:41:25.186935 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:25.186918    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j7qn6"
Jul 10 18:41:25.309358 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:25.309329    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j7qn6"]
Jul 10 18:41:25.314469 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:41:25.314444    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0bfc28_5d5c_42b3_aada_e8f37187da96.slice/crio-3ea23dd33d399022d0f9bfbe20c4682d58f746e9f3e4d0e09375c3ff49810787 WatchSource:0}: Error finding container 3ea23dd33d399022d0f9bfbe20c4682d58f746e9f3e4d0e09375c3ff49810787: Status 404 returned error can't find the container with id 3ea23dd33d399022d0f9bfbe20c4682d58f746e9f3e4d0e09375c3ff49810787
Jul 10 18:41:25.773166 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:25.773132    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j7qn6" event={"ID":"7b0bfc28-5d5c-42b3-aada-e8f37187da96","Type":"ContainerStarted","Data":"3ea23dd33d399022d0f9bfbe20c4682d58f746e9f3e4d0e09375c3ff49810787"}
Jul 10 18:41:26.778042 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:26.778001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j7qn6" event={"ID":"7b0bfc28-5d5c-42b3-aada-e8f37187da96","Type":"ContainerStarted","Data":"9476b20d3f38e20fc66497b260feb7f598ae2c1197c786077904379d08c3dda0"}
Jul 10 18:41:26.778414 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:26.778060    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j7qn6" event={"ID":"7b0bfc28-5d5c-42b3-aada-e8f37187da96","Type":"ContainerStarted","Data":"22ae445c5861c8c51576b942b3f149ba8f9f9420289bbe6a472afa4c0f4fd2c5"}
Jul 10 18:41:26.802004 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:26.801958    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j7qn6" podStartSLOduration=131.014944458 podStartE2EDuration="2m11.80194304s" totalImagesPullingTime="786.998582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:26.79950289 +0000 UTC m=+132.061167674" watchObservedRunningTime="2026-07-10 18:41:26.80194304 +0000 UTC m=+132.063607812"
Jul 10 18:41:31.253500 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:31.253466    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:31.255949 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:31.255916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5f1a2e54-5f36-425c-a557-8b78d8101f1e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rvxcl\" (UID: \"5f1a2e54-5f36-425c-a557-8b78d8101f1e\") " pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:31.420636 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:31.420598    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" podUID="0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 10 18:41:31.537056 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:31.536992    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m5ckx\""
Jul 10 18:41:31.543644 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:31.543628    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rvxcl"
Jul 10 18:41:31.720354 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:31.720322    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rvxcl"]
Jul 10 18:41:31.725706 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:41:31.725676    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1a2e54_5f36_425c_a557_8b78d8101f1e.slice/crio-6be23a2310774899b34be33d95c8a57770011d8c0b744a06cc0dae9eba8b8798 WatchSource:0}: Error finding container 6be23a2310774899b34be33d95c8a57770011d8c0b744a06cc0dae9eba8b8798: Status 404 returned error can't find the container with id 6be23a2310774899b34be33d95c8a57770011d8c0b744a06cc0dae9eba8b8798
Jul 10 18:41:31.793313 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:31.793252    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvxcl" event={"ID":"5f1a2e54-5f36-425c-a557-8b78d8101f1e","Type":"ContainerStarted","Data":"6be23a2310774899b34be33d95c8a57770011d8c0b744a06cc0dae9eba8b8798"}
Jul 10 18:41:35.809258 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:35.809215    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvxcl" event={"ID":"5f1a2e54-5f36-425c-a557-8b78d8101f1e","Type":"ContainerStarted","Data":"beeb5e249117c002da511c5a98a51e8b21c4a613410d4ff837b81f8069724231"}
Jul 10 18:41:36.814842 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:36.814807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvxcl" event={"ID":"5f1a2e54-5f36-425c-a557-8b78d8101f1e","Type":"ContainerStarted","Data":"f5ba4f240033e7a68a7ccf914aa99f89a212f3a5f52d18fd1008383367bd6611"}
Jul 10 18:41:38.822909 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:38.822879    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rvxcl" event={"ID":"5f1a2e54-5f36-425c-a557-8b78d8101f1e","Type":"ContainerStarted","Data":"4db387a7f227c61aa33a3c3630116abdb421dfe1bcd38ae9181320b452aab426"}
Jul 10 18:41:38.849715 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:38.849664    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rvxcl" podStartSLOduration=33.410585957 podStartE2EDuration="39.8496506s" totalImagesPullingTime="6.439064643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 18:40:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:38.848596156 +0000 UTC m=+144.110260930" watchObservedRunningTime="2026-07-10 18:41:38.8496506 +0000 UTC m=+144.111315373"
Jul 10 18:41:41.420432 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:41.420396    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" podUID="0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 10 18:41:41.420875 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:41.420466    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4"
Jul 10 18:41:41.420931 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:41.420904    2558 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"9d55ad82e228ac48650e84a766dd955f32a383c62b3a600adb46e0f04e5eb5d8"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 10 18:41:41.420978 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:41.420954    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" podUID="0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26" containerName="service-proxy" containerID="cri-o://9d55ad82e228ac48650e84a766dd955f32a383c62b3a600adb46e0f04e5eb5d8" gracePeriod=30
Jul 10 18:41:41.833851 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:41.833818    2558 generic.go:356] "Generic (PLEG): container finished" podID="0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26" containerID="9d55ad82e228ac48650e84a766dd955f32a383c62b3a600adb46e0f04e5eb5d8" exitCode=2
Jul 10 18:41:41.834018 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:41.833896    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" event={"ID":"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26","Type":"ContainerDied","Data":"9d55ad82e228ac48650e84a766dd955f32a383c62b3a600adb46e0f04e5eb5d8"}
Jul 10 18:41:41.834018 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:41.833954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-68d6f475bc-mmcz4" event={"ID":"0cf52bbe-5f33-47b6-b0b3-49e6d8ab7c26","Type":"ContainerStarted","Data":"ec76249f83c638d30cd7fafd433987a690747433f9207cec1cc4f89fd7acf2b3"}
Jul 10 18:41:46.825826 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.825799    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g8dqw"]
Jul 10 18:41:46.827966 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.827950    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.830172 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.830153    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 18:41:46.830292 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.830219    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 18:41:46.830484 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.830463    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 18:41:46.830585 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.830472    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 18:41:46.830585 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.830564    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6r84z\""
Jul 10 18:41:46.831426 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.831411    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 18:41:46.831480 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.831444    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 18:41:46.969648 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969608    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-tls\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.969648 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-root\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.969901 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d75b631-1c18-44f7-8893-0f703c28900d-metrics-client-ca\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.969901 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969696    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-sys\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.969901 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-wtmp\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.969901 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969827    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsskc\" (UniqueName: \"kubernetes.io/projected/2d75b631-1c18-44f7-8893-0f703c28900d-kube-api-access-gsskc\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.969901 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969862    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.969901 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969896    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:46.970132 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:46.969933    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-textfile\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070227 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-wtmp\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070393 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsskc\" (UniqueName: \"kubernetes.io/projected/2d75b631-1c18-44f7-8893-0f703c28900d-kube-api-access-gsskc\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070393 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070393 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070393 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-textfile\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070393 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-tls\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-wtmp\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-root\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070445    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d75b631-1c18-44f7-8893-0f703c28900d-metrics-client-ca\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-root\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-sys\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:47.070495    2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d75b631-1c18-44f7-8893-0f703c28900d-sys\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.070628 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:47.070596    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-tls podName:2d75b631-1c18-44f7-8893-0f703c28900d nodeName:}" failed. No retries permitted until 2026-07-10 18:41:47.570573359 +0000 UTC m=+152.832238147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-tls") pod "node-exporter-g8dqw" (UID: "2d75b631-1c18-44f7-8893-0f703c28900d") : secret "node-exporter-tls" not found
Jul 10 18:41:47.070928 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.070732    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-textfile\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.071061 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.071037    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d75b631-1c18-44f7-8893-0f703c28900d-metrics-client-ca\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.071119 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.071040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-accelerators-collector-config\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.072715 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.072698    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.082876 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.082821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsskc\" (UniqueName: \"kubernetes.io/projected/2d75b631-1c18-44f7-8893-0f703c28900d-kube-api-access-gsskc\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.575172 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.575136    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-tls\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.577337 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.577309    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d75b631-1c18-44f7-8893-0f703c28900d-node-exporter-tls\") pod \"node-exporter-g8dqw\" (UID: \"2d75b631-1c18-44f7-8893-0f703c28900d\") " pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.736670 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.736638    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g8dqw"
Jul 10 18:41:47.745544 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:41:47.745518    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d75b631_1c18_44f7_8893_0f703c28900d.slice/crio-ba1883ca6d702b8ffb256a8d360b8cabc877c95a8bedd1d129e9b31f50094a17 WatchSource:0}: Error finding container ba1883ca6d702b8ffb256a8d360b8cabc877c95a8bedd1d129e9b31f50094a17: Status 404 returned error can't find the container with id ba1883ca6d702b8ffb256a8d360b8cabc877c95a8bedd1d129e9b31f50094a17
Jul 10 18:41:47.853533 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:47.853460    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8dqw" event={"ID":"2d75b631-1c18-44f7-8893-0f703c28900d","Type":"ContainerStarted","Data":"ba1883ca6d702b8ffb256a8d360b8cabc877c95a8bedd1d129e9b31f50094a17"}
Jul 10 18:41:48.857908 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:48.857879    2558 generic.go:356] "Generic (PLEG): container finished" podID="2d75b631-1c18-44f7-8893-0f703c28900d" containerID="9ddf7849b7a8c9eecc37e995cdf80ca4fae1b44c64a5e75ee85c348d1c32d1ec" exitCode=0
Jul 10 18:41:48.858275 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:48.857973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8dqw" event={"ID":"2d75b631-1c18-44f7-8893-0f703c28900d","Type":"ContainerDied","Data":"9ddf7849b7a8c9eecc37e995cdf80ca4fae1b44c64a5e75ee85c348d1c32d1ec"}
Jul 10 18:41:49.863518 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:49.863476    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8dqw" event={"ID":"2d75b631-1c18-44f7-8893-0f703c28900d","Type":"ContainerStarted","Data":"99332a8b9a78af701924e2784743b949d3b92ce7bbb87b8970188c5a22ee419d"}
Jul 10 18:41:49.863911 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:49.863536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g8dqw" event={"ID":"2d75b631-1c18-44f7-8893-0f703c28900d","Type":"ContainerStarted","Data":"cab88278cc2ed3fc29832915742bde2cb5ff8c0d33fc4ab59784662cf80d4de2"}
Jul 10 18:41:49.886684 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:49.886635    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g8dqw" podStartSLOduration=2.901621952 podStartE2EDuration="3.88661825s" totalImagesPullingTime="921.854695ms" totalInitContainerRuntime="63.141603ms" isStatefulPod=true podCreationTimestamp="2026-07-10 18:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:41:49.884802908 +0000 UTC m=+155.146467681" watchObservedRunningTime="2026-07-10 18:41:49.88661825 +0000 UTC m=+155.148283023"
Jul 10 18:41:54.103914 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:54.103858    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f56675b7f-v8np5" podUID="6df8b103-18a2-427a-bb8e-815f73a5ebdb"
Jul 10 18:41:54.153422 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:54.153385    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hhxbq" podUID="ef7e1cdc-4e50-4260-9838-df169c0b9ade"
Jul 10 18:41:54.758938 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:54.758904    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f56675b7f-v8np5"]
Jul 10 18:41:54.879399 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:54.879371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:41:54.879544 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:54.879371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:41:54.879764 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:41:54.879724    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6f56675b7f-v8np5" podUID="6df8b103-18a2-427a-bb8e-815f73a5ebdb"
Jul 10 18:41:55.882119 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.882090    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:41:55.887394 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.887371    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:41:55.937176 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937145    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-image-registry-private-configuration\") pod \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") "
Jul 10 18:41:55.937333 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937188    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-bound-sa-token\") pod \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") "
Jul 10 18:41:55.937333 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937207    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-certificates\") pod \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") "
Jul 10 18:41:55.937333 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937228    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-trusted-ca\") pod \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") "
Jul 10 18:41:55.937333 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937257    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-installation-pull-secrets\") pod \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") "
Jul 10 18:41:55.937333 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937292    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpzqj\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-kube-api-access-rpzqj\") pod \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") "
Jul 10 18:41:55.937588 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937339    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6df8b103-18a2-427a-bb8e-815f73a5ebdb-ca-trust-extracted\") pod \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\" (UID: \"6df8b103-18a2-427a-bb8e-815f73a5ebdb\") "
Jul 10 18:41:55.937826 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937679    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6df8b103-18a2-427a-bb8e-815f73a5ebdb" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:41:55.937929 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937721    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6df8b103-18a2-427a-bb8e-815f73a5ebdb" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:41:55.937929 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.937776    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df8b103-18a2-427a-bb8e-815f73a5ebdb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6df8b103-18a2-427a-bb8e-815f73a5ebdb" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:41:55.939528 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.939496    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6df8b103-18a2-427a-bb8e-815f73a5ebdb" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:41:55.939640 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.939545    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-kube-api-access-rpzqj" (OuterVolumeSpecName: "kube-api-access-rpzqj") pod "6df8b103-18a2-427a-bb8e-815f73a5ebdb" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb"). InnerVolumeSpecName "kube-api-access-rpzqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:41:55.939640 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.939577    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6df8b103-18a2-427a-bb8e-815f73a5ebdb" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:41:55.939640 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:55.939600    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6df8b103-18a2-427a-bb8e-815f73a5ebdb" (UID: "6df8b103-18a2-427a-bb8e-815f73a5ebdb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:41:56.038065 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.038033    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpzqj\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-kube-api-access-rpzqj\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:56.038065 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.038058    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6df8b103-18a2-427a-bb8e-815f73a5ebdb-ca-trust-extracted\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:56.038065 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.038067    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-image-registry-private-configuration\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:56.038255 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.038077    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-bound-sa-token\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:56.038255 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.038085    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-certificates\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:56.038255 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.038093    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6df8b103-18a2-427a-bb8e-815f73a5ebdb-trusted-ca\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:56.038255 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.038101    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6df8b103-18a2-427a-bb8e-815f73a5ebdb-installation-pull-secrets\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:56.888985 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.886435    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f56675b7f-v8np5"
Jul 10 18:41:56.925277 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.925244    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f56675b7f-v8np5"]
Jul 10 18:41:56.929733 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.929704    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f56675b7f-v8np5"]
Jul 10 18:41:56.944983 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:56.944952    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6df8b103-18a2-427a-bb8e-815f73a5ebdb-registry-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:41:57.256368 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:57.256334    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6df8b103-18a2-427a-bb8e-815f73a5ebdb" path="/var/lib/kubelet/pods/6df8b103-18a2-427a-bb8e-815f73a5ebdb/volumes"
Jul 10 18:41:59.061704 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:59.061672    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:41:59.063942 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:59.063922    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef7e1cdc-4e50-4260-9838-df169c0b9ade-metrics-tls\") pod \"dns-default-hhxbq\" (UID: \"ef7e1cdc-4e50-4260-9838-df169c0b9ade\") " pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:41:59.084521 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:59.084500    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s4tdx\""
Jul 10 18:41:59.090695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:59.090679    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:41:59.217975 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:59.217945    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hhxbq"]
Jul 10 18:41:59.221035 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:41:59.221009    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7e1cdc_4e50_4260_9838_df169c0b9ade.slice/crio-ed939f6ca3106d6ebc2d207d1f89141ba899809244b62c1f21b10af6e3079e8b WatchSource:0}: Error finding container ed939f6ca3106d6ebc2d207d1f89141ba899809244b62c1f21b10af6e3079e8b: Status 404 returned error can't find the container with id ed939f6ca3106d6ebc2d207d1f89141ba899809244b62c1f21b10af6e3079e8b
Jul 10 18:41:59.896822 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:41:59.896782    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hhxbq" event={"ID":"ef7e1cdc-4e50-4260-9838-df169c0b9ade","Type":"ContainerStarted","Data":"ed939f6ca3106d6ebc2d207d1f89141ba899809244b62c1f21b10af6e3079e8b"}
Jul 10 18:42:01.904462 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:42:01.904419    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hhxbq" event={"ID":"ef7e1cdc-4e50-4260-9838-df169c0b9ade","Type":"ContainerStarted","Data":"c74418f72dda37e0eef02268751ee0f9292a76ffc2f58b21002c9d02e9c1e8c6"}
Jul 10 18:42:01.904904 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:42:01.904481    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hhxbq" event={"ID":"ef7e1cdc-4e50-4260-9838-df169c0b9ade","Type":"ContainerStarted","Data":"fc21ada073f01b19682dfc5b08987614bf781fde75e4c0fff918805182d1f991"}
Jul 10 18:42:01.904904 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:42:01.904543    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:42:01.932505 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:42:01.932448    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hhxbq" podStartSLOduration=129.266681232 podStartE2EDuration="2m10.932429849s" totalImagesPullingTime="1.665748617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:42:01.929908521 +0000 UTC m=+167.191573294" watchObservedRunningTime="2026-07-10 18:42:01.932429849 +0000 UTC m=+167.194094667"
Jul 10 18:42:04.771262 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:42:04.771227    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xr5nv_9f070177-34ce-4586-94ed-96206e46da76/serve-healthcheck-canary/0.log"
Jul 10 18:42:11.910012 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:42:11.909930    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hhxbq"
Jul 10 18:44:15.168083 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:44:15.168049    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:44:15.171404 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:44:15.171385    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 18:44:15.173614 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:44:15.173598    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:46:30.613235 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.613204    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-659578d74d-pkm5q"]
Jul 10 18:46:30.616096 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.616080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.618631 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.618611    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 10 18:46:30.619002 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.618985    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 10 18:46:30.619332 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.619309    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-s2dms\""
Jul 10 18:46:30.619426 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.619338    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 10 18:46:30.628530 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.628507    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-pkm5q"]
Jul 10 18:46:30.660840 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.660808    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-vhtrw"]
Jul 10 18:46:30.664088 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.664070    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:30.666512 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.666469    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 10 18:46:30.666774 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.666760    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-w4q5z\""
Jul 10 18:46:30.677142 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.677121    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vhtrw"]
Jul 10 18:46:30.707719 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.707694    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6zn\" (UniqueName: \"kubernetes.io/projected/479937a6-77d6-4506-b671-22f1874a25fc-kube-api-access-gn6zn\") pod \"kserve-controller-manager-659578d74d-pkm5q\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") " pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.707831 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.707736    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/479937a6-77d6-4506-b671-22f1874a25fc-cert\") pod \"kserve-controller-manager-659578d74d-pkm5q\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") " pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.808905 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.808873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/479937a6-77d6-4506-b671-22f1874a25fc-cert\") pod \"kserve-controller-manager-659578d74d-pkm5q\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") " pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.809079 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.808915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5mb\" (UniqueName: \"kubernetes.io/projected/d790a2c2-0ccf-4e95-826a-6f3adc27de39-kube-api-access-kv5mb\") pod \"seaweedfs-64568bcd49-vhtrw\" (UID: \"d790a2c2-0ccf-4e95-826a-6f3adc27de39\") " pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:30.809079 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.809041    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d790a2c2-0ccf-4e95-826a-6f3adc27de39-data\") pod \"seaweedfs-64568bcd49-vhtrw\" (UID: \"d790a2c2-0ccf-4e95-826a-6f3adc27de39\") " pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:30.809079 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.809079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6zn\" (UniqueName: \"kubernetes.io/projected/479937a6-77d6-4506-b671-22f1874a25fc-kube-api-access-gn6zn\") pod \"kserve-controller-manager-659578d74d-pkm5q\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") " pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.811323 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.811303    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/479937a6-77d6-4506-b671-22f1874a25fc-cert\") pod \"kserve-controller-manager-659578d74d-pkm5q\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") " pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.828095 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.828075    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6zn\" (UniqueName: \"kubernetes.io/projected/479937a6-77d6-4506-b671-22f1874a25fc-kube-api-access-gn6zn\") pod \"kserve-controller-manager-659578d74d-pkm5q\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") " pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.909809 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.909693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d790a2c2-0ccf-4e95-826a-6f3adc27de39-data\") pod \"seaweedfs-64568bcd49-vhtrw\" (UID: \"d790a2c2-0ccf-4e95-826a-6f3adc27de39\") " pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:30.909809 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.909786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5mb\" (UniqueName: \"kubernetes.io/projected/d790a2c2-0ccf-4e95-826a-6f3adc27de39-kube-api-access-kv5mb\") pod \"seaweedfs-64568bcd49-vhtrw\" (UID: \"d790a2c2-0ccf-4e95-826a-6f3adc27de39\") " pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:30.910084 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.910061    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d790a2c2-0ccf-4e95-826a-6f3adc27de39-data\") pod \"seaweedfs-64568bcd49-vhtrw\" (UID: \"d790a2c2-0ccf-4e95-826a-6f3adc27de39\") " pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:30.919052 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.919026    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5mb\" (UniqueName: \"kubernetes.io/projected/d790a2c2-0ccf-4e95-826a-6f3adc27de39-kube-api-access-kv5mb\") pod \"seaweedfs-64568bcd49-vhtrw\" (UID: \"d790a2c2-0ccf-4e95-826a-6f3adc27de39\") " pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:30.925853 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.925826    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:30.972805 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:30.972694    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:31.061169 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:31.061141    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-pkm5q"]
Jul 10 18:46:31.063641 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:46:31.063596    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479937a6_77d6_4506_b671_22f1874a25fc.slice/crio-b54c153b19dea1811b8339c9b36a438fe75263f714a2d6cff5b09247fd96b57a WatchSource:0}: Error finding container b54c153b19dea1811b8339c9b36a438fe75263f714a2d6cff5b09247fd96b57a: Status 404 returned error can't find the container with id b54c153b19dea1811b8339c9b36a438fe75263f714a2d6cff5b09247fd96b57a
Jul 10 18:46:31.065040 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:31.065019    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:46:31.104228 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:31.104141    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-vhtrw"]
Jul 10 18:46:31.106462 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:46:31.106440    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd790a2c2_0ccf_4e95_826a_6f3adc27de39.slice/crio-20e452da371dfc627568088da8d75d5aa8adc80dcb2396c3dc9f2a673fad6d9d WatchSource:0}: Error finding container 20e452da371dfc627568088da8d75d5aa8adc80dcb2396c3dc9f2a673fad6d9d: Status 404 returned error can't find the container with id 20e452da371dfc627568088da8d75d5aa8adc80dcb2396c3dc9f2a673fad6d9d
Jul 10 18:46:31.725500 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:31.725459    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vhtrw" event={"ID":"d790a2c2-0ccf-4e95-826a-6f3adc27de39","Type":"ContainerStarted","Data":"20e452da371dfc627568088da8d75d5aa8adc80dcb2396c3dc9f2a673fad6d9d"}
Jul 10 18:46:31.727136 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:31.727043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-pkm5q" event={"ID":"479937a6-77d6-4506-b671-22f1874a25fc","Type":"ContainerStarted","Data":"b54c153b19dea1811b8339c9b36a438fe75263f714a2d6cff5b09247fd96b57a"}
Jul 10 18:46:35.742844 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:35.742809    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-vhtrw" event={"ID":"d790a2c2-0ccf-4e95-826a-6f3adc27de39","Type":"ContainerStarted","Data":"a817eda7ea44a9c4464623bfa36de0b882168b39e7942ebebccc8d941ed05498"}
Jul 10 18:46:35.743302 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:35.742923    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:46:35.744290 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:35.744264    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-pkm5q" event={"ID":"479937a6-77d6-4506-b671-22f1874a25fc","Type":"ContainerStarted","Data":"768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e"}
Jul 10 18:46:35.744415 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:35.744364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:46:35.759547 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:35.759506    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-vhtrw" podStartSLOduration=1.990205366 podStartE2EDuration="5.759493628s" totalImagesPullingTime="3.769288262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:46:35.758534658 +0000 UTC m=+441.020199430" watchObservedRunningTime="2026-07-10 18:46:35.759493628 +0000 UTC m=+441.021158401"
Jul 10 18:46:35.779102 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:35.775148    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-659578d74d-pkm5q" podStartSLOduration=2.049706634 podStartE2EDuration="5.775131011s" totalImagesPullingTime="3.725424377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:46:35.773308854 +0000 UTC m=+441.034973628" watchObservedRunningTime="2026-07-10 18:46:35.775131011 +0000 UTC m=+441.036795784"
Jul 10 18:46:41.749442 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:46:41.749371    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-vhtrw"
Jul 10 18:47:06.174903 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.174866    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-pkm5q"]
Jul 10 18:47:06.175392 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.175130    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-659578d74d-pkm5q" podUID="479937a6-77d6-4506-b671-22f1874a25fc" containerName="manager" containerID="cri-o://768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e" gracePeriod=10
Jul 10 18:47:06.180164 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.180144    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:47:06.201940 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.201911    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-659578d74d-nt8hp"]
Jul 10 18:47:06.203721 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.203705    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.212850 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.212828    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-nt8hp"]
Jul 10 18:47:06.255718 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.255691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2329682-ee12-4c91-89f9-b5bf5fee66fa-cert\") pod \"kserve-controller-manager-659578d74d-nt8hp\" (UID: \"c2329682-ee12-4c91-89f9-b5bf5fee66fa\") " pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.255858 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.255820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxz7\" (UniqueName: \"kubernetes.io/projected/c2329682-ee12-4c91-89f9-b5bf5fee66fa-kube-api-access-hhxz7\") pod \"kserve-controller-manager-659578d74d-nt8hp\" (UID: \"c2329682-ee12-4c91-89f9-b5bf5fee66fa\") " pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.356263 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.356234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2329682-ee12-4c91-89f9-b5bf5fee66fa-cert\") pod \"kserve-controller-manager-659578d74d-nt8hp\" (UID: \"c2329682-ee12-4c91-89f9-b5bf5fee66fa\") " pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.356403 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.356302    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxz7\" (UniqueName: \"kubernetes.io/projected/c2329682-ee12-4c91-89f9-b5bf5fee66fa-kube-api-access-hhxz7\") pod \"kserve-controller-manager-659578d74d-nt8hp\" (UID: \"c2329682-ee12-4c91-89f9-b5bf5fee66fa\") " pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.359230 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.359201    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2329682-ee12-4c91-89f9-b5bf5fee66fa-cert\") pod \"kserve-controller-manager-659578d74d-nt8hp\" (UID: \"c2329682-ee12-4c91-89f9-b5bf5fee66fa\") " pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.365130 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.365103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxz7\" (UniqueName: \"kubernetes.io/projected/c2329682-ee12-4c91-89f9-b5bf5fee66fa-kube-api-access-hhxz7\") pod \"kserve-controller-manager-659578d74d-nt8hp\" (UID: \"c2329682-ee12-4c91-89f9-b5bf5fee66fa\") " pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.408362 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.408342    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:47:06.457608 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.457545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn6zn\" (UniqueName: \"kubernetes.io/projected/479937a6-77d6-4506-b671-22f1874a25fc-kube-api-access-gn6zn\") pod \"479937a6-77d6-4506-b671-22f1874a25fc\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") "
Jul 10 18:47:06.457608 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.457590    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/479937a6-77d6-4506-b671-22f1874a25fc-cert\") pod \"479937a6-77d6-4506-b671-22f1874a25fc\" (UID: \"479937a6-77d6-4506-b671-22f1874a25fc\") "
Jul 10 18:47:06.459562 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.459541    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479937a6-77d6-4506-b671-22f1874a25fc-cert" (OuterVolumeSpecName: "cert") pod "479937a6-77d6-4506-b671-22f1874a25fc" (UID: "479937a6-77d6-4506-b671-22f1874a25fc"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:47:06.459659 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.459604    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479937a6-77d6-4506-b671-22f1874a25fc-kube-api-access-gn6zn" (OuterVolumeSpecName: "kube-api-access-gn6zn") pod "479937a6-77d6-4506-b671-22f1874a25fc" (UID: "479937a6-77d6-4506-b671-22f1874a25fc"). InnerVolumeSpecName "kube-api-access-gn6zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:47:06.558182 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.558145    2558 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/479937a6-77d6-4506-b671-22f1874a25fc-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:47:06.558182 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.558177    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6zn\" (UniqueName: \"kubernetes.io/projected/479937a6-77d6-4506-b671-22f1874a25fc-kube-api-access-gn6zn\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:47:06.561182 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.561161    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:06.682296 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.682243    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-nt8hp"]
Jul 10 18:47:06.685774 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:47:06.685735    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2329682_ee12_4c91_89f9_b5bf5fee66fa.slice/crio-69a4599f06a5629697db96377d844743e6a9ed9ad80781d1c068ee2734c3af15 WatchSource:0}: Error finding container 69a4599f06a5629697db96377d844743e6a9ed9ad80781d1c068ee2734c3af15: Status 404 returned error can't find the container with id 69a4599f06a5629697db96377d844743e6a9ed9ad80781d1c068ee2734c3af15
Jul 10 18:47:06.847243 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.847207    2558 generic.go:356] "Generic (PLEG): container finished" podID="479937a6-77d6-4506-b671-22f1874a25fc" containerID="768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e" exitCode=0
Jul 10 18:47:06.847419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.847257    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-659578d74d-pkm5q"
Jul 10 18:47:06.847419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.847287    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-pkm5q" event={"ID":"479937a6-77d6-4506-b671-22f1874a25fc","Type":"ContainerDied","Data":"768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e"}
Jul 10 18:47:06.847419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.847345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-pkm5q" event={"ID":"479937a6-77d6-4506-b671-22f1874a25fc","Type":"ContainerDied","Data":"b54c153b19dea1811b8339c9b36a438fe75263f714a2d6cff5b09247fd96b57a"}
Jul 10 18:47:06.847419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.847376    2558 scope.go:117] "RemoveContainer" containerID="768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e"
Jul 10 18:47:06.848572 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.848443    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-nt8hp" event={"ID":"c2329682-ee12-4c91-89f9-b5bf5fee66fa","Type":"ContainerStarted","Data":"69a4599f06a5629697db96377d844743e6a9ed9ad80781d1c068ee2734c3af15"}
Jul 10 18:47:06.855780 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.855762    2558 scope.go:117] "RemoveContainer" containerID="768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e"
Jul 10 18:47:06.856092 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:47:06.856072    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e\": container with ID starting with 768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e not found: ID does not exist" containerID="768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e"
Jul 10 18:47:06.856139 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.856103    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e"} err="failed to get container status \"768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e\": rpc error: code = NotFound desc = could not find container \"768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e\": container with ID starting with 768ae5420a44c0612059262b13ccd985d217bdd7c0c2783781bbcb29a7f30c2e not found: ID does not exist"
Jul 10 18:47:06.868277 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.868253    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-pkm5q"]
Jul 10 18:47:06.871497 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:06.871479    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-659578d74d-pkm5q"]
Jul 10 18:47:07.256172 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:07.256141    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="479937a6-77d6-4506-b671-22f1874a25fc" path="/var/lib/kubelet/pods/479937a6-77d6-4506-b671-22f1874a25fc/volumes"
Jul 10 18:47:07.853382 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:07.853350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-659578d74d-nt8hp" event={"ID":"c2329682-ee12-4c91-89f9-b5bf5fee66fa","Type":"ContainerStarted","Data":"16f893ad1052996e846203806f326bdd040299fdb10131f8011792d2b869c494"}
Jul 10 18:47:07.853585 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:07.853415    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:07.870161 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:07.870087    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-659578d74d-nt8hp" podStartSLOduration=1.427333763 podStartE2EDuration="1.870068682s" totalImagesPullingTime="442.734919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:07.869464002 +0000 UTC m=+473.131128775" watchObservedRunningTime="2026-07-10 18:47:07.870068682 +0000 UTC m=+473.131733457"
Jul 10 18:47:38.862524 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:38.862493    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-659578d74d-nt8hp"
Jul 10 18:47:39.861075 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.861042    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b5kp2"]
Jul 10 18:47:39.861365 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.861349    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479937a6-77d6-4506-b671-22f1874a25fc" containerName="manager"
Jul 10 18:47:39.861448 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.861368    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="479937a6-77d6-4506-b671-22f1874a25fc" containerName="manager"
Jul 10 18:47:39.861505 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.861451    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="479937a6-77d6-4506-b671-22f1874a25fc" containerName="manager"
Jul 10 18:47:39.864017 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.863996    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:39.867801 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.867783    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 10 18:47:39.868158 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.868140    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-825zx\""
Jul 10 18:47:39.877376 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.877355    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b5kp2"]
Jul 10 18:47:39.986641 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.986608    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c8xk\" (UniqueName: \"kubernetes.io/projected/54270212-d6b3-4e62-b52a-74985d4dcd57-kube-api-access-5c8xk\") pod \"odh-model-controller-84b54b7f97-b5kp2\" (UID: \"54270212-d6b3-4e62-b52a-74985d4dcd57\") " pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:39.986641 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:39.986642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54270212-d6b3-4e62-b52a-74985d4dcd57-cert\") pod \"odh-model-controller-84b54b7f97-b5kp2\" (UID: \"54270212-d6b3-4e62-b52a-74985d4dcd57\") " pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:40.087998 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.087966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c8xk\" (UniqueName: \"kubernetes.io/projected/54270212-d6b3-4e62-b52a-74985d4dcd57-kube-api-access-5c8xk\") pod \"odh-model-controller-84b54b7f97-b5kp2\" (UID: \"54270212-d6b3-4e62-b52a-74985d4dcd57\") " pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:40.087998 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.087997    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54270212-d6b3-4e62-b52a-74985d4dcd57-cert\") pod \"odh-model-controller-84b54b7f97-b5kp2\" (UID: \"54270212-d6b3-4e62-b52a-74985d4dcd57\") " pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:40.088212 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:47:40.088091    2558 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 10 18:47:40.088212 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:47:40.088164    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54270212-d6b3-4e62-b52a-74985d4dcd57-cert podName:54270212-d6b3-4e62-b52a-74985d4dcd57 nodeName:}" failed. No retries permitted until 2026-07-10 18:47:40.588137929 +0000 UTC m=+505.849802696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/54270212-d6b3-4e62-b52a-74985d4dcd57-cert") pod "odh-model-controller-84b54b7f97-b5kp2" (UID: "54270212-d6b3-4e62-b52a-74985d4dcd57") : secret "odh-model-controller-webhook-cert" not found
Jul 10 18:47:40.099180 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.099147    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c8xk\" (UniqueName: \"kubernetes.io/projected/54270212-d6b3-4e62-b52a-74985d4dcd57-kube-api-access-5c8xk\") pod \"odh-model-controller-84b54b7f97-b5kp2\" (UID: \"54270212-d6b3-4e62-b52a-74985d4dcd57\") " pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:40.591508 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.591462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54270212-d6b3-4e62-b52a-74985d4dcd57-cert\") pod \"odh-model-controller-84b54b7f97-b5kp2\" (UID: \"54270212-d6b3-4e62-b52a-74985d4dcd57\") " pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:40.593841 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.593821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54270212-d6b3-4e62-b52a-74985d4dcd57-cert\") pod \"odh-model-controller-84b54b7f97-b5kp2\" (UID: \"54270212-d6b3-4e62-b52a-74985d4dcd57\") " pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:40.773943 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.773893    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:40.895168 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.895139    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-b5kp2"]
Jul 10 18:47:40.898495 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:47:40.898469    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54270212_d6b3_4e62_b52a_74985d4dcd57.slice/crio-1d3e0c0ed43d8adfb43468c15bf6bbc0db4a322380dc0355f2193f2ba0af13ff WatchSource:0}: Error finding container 1d3e0c0ed43d8adfb43468c15bf6bbc0db4a322380dc0355f2193f2ba0af13ff: Status 404 returned error can't find the container with id 1d3e0c0ed43d8adfb43468c15bf6bbc0db4a322380dc0355f2193f2ba0af13ff
Jul 10 18:47:40.961112 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:40.961078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b5kp2" event={"ID":"54270212-d6b3-4e62-b52a-74985d4dcd57","Type":"ContainerStarted","Data":"1d3e0c0ed43d8adfb43468c15bf6bbc0db4a322380dc0355f2193f2ba0af13ff"}
Jul 10 18:47:43.973679 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:43.973594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-b5kp2" event={"ID":"54270212-d6b3-4e62-b52a-74985d4dcd57","Type":"ContainerStarted","Data":"395480eba2389ae2be8a777269ac35b9560c36c4d971aa3a17bded4dc00de841"}
Jul 10 18:47:43.974066 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:43.973755    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:44.006663 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:44.006612    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-b5kp2" podStartSLOduration=2.342238229 podStartE2EDuration="5.006595688s" totalImagesPullingTime="2.664357459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:47:44.005736172 +0000 UTC m=+509.267400947" watchObservedRunningTime="2026-07-10 18:47:44.006595688 +0000 UTC m=+509.268260461"
Jul 10 18:47:54.978665 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:54.978634    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-b5kp2"
Jul 10 18:47:55.779691 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:55.779655    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-jp47q"]
Jul 10 18:47:55.782826 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:55.782806    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jp47q"
Jul 10 18:47:55.789814 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:55.789795    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jp47q"]
Jul 10 18:47:55.908145 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:55.908114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqns\" (UniqueName: \"kubernetes.io/projected/80e5d68b-f539-44c6-95a9-cbc49d705fc3-kube-api-access-ttqns\") pod \"s3-init-jp47q\" (UID: \"80e5d68b-f539-44c6-95a9-cbc49d705fc3\") " pod="kserve/s3-init-jp47q"
Jul 10 18:47:56.009380 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:56.009346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqns\" (UniqueName: \"kubernetes.io/projected/80e5d68b-f539-44c6-95a9-cbc49d705fc3-kube-api-access-ttqns\") pod \"s3-init-jp47q\" (UID: \"80e5d68b-f539-44c6-95a9-cbc49d705fc3\") " pod="kserve/s3-init-jp47q"
Jul 10 18:47:56.017804 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:56.017773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqns\" (UniqueName: \"kubernetes.io/projected/80e5d68b-f539-44c6-95a9-cbc49d705fc3-kube-api-access-ttqns\") pod \"s3-init-jp47q\" (UID: \"80e5d68b-f539-44c6-95a9-cbc49d705fc3\") " pod="kserve/s3-init-jp47q"
Jul 10 18:47:56.091841 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:56.091762    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jp47q"
Jul 10 18:47:56.213842 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:56.213755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-jp47q"]
Jul 10 18:47:56.216169 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:47:56.216138    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e5d68b_f539_44c6_95a9_cbc49d705fc3.slice/crio-450707c50443b549257dfaa2a8998bcdbf1fa8e3a387060d5f28ed4c00097af8 WatchSource:0}: Error finding container 450707c50443b549257dfaa2a8998bcdbf1fa8e3a387060d5f28ed4c00097af8: Status 404 returned error can't find the container with id 450707c50443b549257dfaa2a8998bcdbf1fa8e3a387060d5f28ed4c00097af8
Jul 10 18:47:57.028838 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:47:57.028778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jp47q" event={"ID":"80e5d68b-f539-44c6-95a9-cbc49d705fc3","Type":"ContainerStarted","Data":"450707c50443b549257dfaa2a8998bcdbf1fa8e3a387060d5f28ed4c00097af8"}
Jul 10 18:48:02.048666 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:02.048630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jp47q" event={"ID":"80e5d68b-f539-44c6-95a9-cbc49d705fc3","Type":"ContainerStarted","Data":"2b16faf170360da3c39f5db2d0f18cee6e05fef1606c201a67fe561c6a0301a5"}
Jul 10 18:48:02.065059 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:02.065001    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-jp47q" podStartSLOduration=2.264945734 podStartE2EDuration="7.064983918s" totalImagesPullingTime="4.800038184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:02.064183309 +0000 UTC m=+527.325848080" watchObservedRunningTime="2026-07-10 18:48:02.064983918 +0000 UTC m=+527.326648692"
Jul 10 18:48:04.056263 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:04.056230    2558 generic.go:356] "Generic (PLEG): container finished" podID="80e5d68b-f539-44c6-95a9-cbc49d705fc3" containerID="2b16faf170360da3c39f5db2d0f18cee6e05fef1606c201a67fe561c6a0301a5" exitCode=0
Jul 10 18:48:04.056654 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:04.056269    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jp47q" event={"ID":"80e5d68b-f539-44c6-95a9-cbc49d705fc3","Type":"ContainerDied","Data":"2b16faf170360da3c39f5db2d0f18cee6e05fef1606c201a67fe561c6a0301a5"}
Jul 10 18:48:05.184342 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:05.184320    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jp47q"
Jul 10 18:48:05.282161 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:05.282131    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttqns\" (UniqueName: \"kubernetes.io/projected/80e5d68b-f539-44c6-95a9-cbc49d705fc3-kube-api-access-ttqns\") pod \"80e5d68b-f539-44c6-95a9-cbc49d705fc3\" (UID: \"80e5d68b-f539-44c6-95a9-cbc49d705fc3\") "
Jul 10 18:48:05.284099 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:05.284061    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e5d68b-f539-44c6-95a9-cbc49d705fc3-kube-api-access-ttqns" (OuterVolumeSpecName: "kube-api-access-ttqns") pod "80e5d68b-f539-44c6-95a9-cbc49d705fc3" (UID: "80e5d68b-f539-44c6-95a9-cbc49d705fc3"). InnerVolumeSpecName "kube-api-access-ttqns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:48:05.382788 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:05.382686    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttqns\" (UniqueName: \"kubernetes.io/projected/80e5d68b-f539-44c6-95a9-cbc49d705fc3-kube-api-access-ttqns\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:48:06.063082 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.063046    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-jp47q" event={"ID":"80e5d68b-f539-44c6-95a9-cbc49d705fc3","Type":"ContainerDied","Data":"450707c50443b549257dfaa2a8998bcdbf1fa8e3a387060d5f28ed4c00097af8"}
Jul 10 18:48:06.063269 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.063093    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="450707c50443b549257dfaa2a8998bcdbf1fa8e3a387060d5f28ed4c00097af8"
Jul 10 18:48:06.063269 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.063061    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-jp47q"
Jul 10 18:48:06.858921 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.858889    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"]
Jul 10 18:48:06.859266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.859140    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e5d68b-f539-44c6-95a9-cbc49d705fc3" containerName="s3-init"
Jul 10 18:48:06.859266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.859151    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e5d68b-f539-44c6-95a9-cbc49d705fc3" containerName="s3-init"
Jul 10 18:48:06.859266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.859206    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e5d68b-f539-44c6-95a9-cbc49d705fc3" containerName="s3-init"
Jul 10 18:48:06.861453 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.861438    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:06.863594 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.863570    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 10 18:48:06.869873 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.869852    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"]
Jul 10 18:48:06.992904 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.992874    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdb9x\" (UniqueName: \"kubernetes.io/projected/9637a42f-c05d-4370-9c49-a5d235015f92-kube-api-access-tdb9x\") pod \"seaweedfs-tls-custom-6644bf5d6c-mz5nv\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:06.993025 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:06.992912    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9637a42f-c05d-4370-9c49-a5d235015f92-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mz5nv\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:07.093486 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:07.093457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdb9x\" (UniqueName: \"kubernetes.io/projected/9637a42f-c05d-4370-9c49-a5d235015f92-kube-api-access-tdb9x\") pod \"seaweedfs-tls-custom-6644bf5d6c-mz5nv\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:07.093486 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:07.093491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9637a42f-c05d-4370-9c49-a5d235015f92-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mz5nv\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:07.093933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:07.093913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9637a42f-c05d-4370-9c49-a5d235015f92-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-mz5nv\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:07.102508 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:07.102485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdb9x\" (UniqueName: \"kubernetes.io/projected/9637a42f-c05d-4370-9c49-a5d235015f92-kube-api-access-tdb9x\") pod \"seaweedfs-tls-custom-6644bf5d6c-mz5nv\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:07.170770 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:07.170692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:07.297394 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:07.297365    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"]
Jul 10 18:48:07.300489 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:48:07.300465    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9637a42f_c05d_4370_9c49_a5d235015f92.slice/crio-097858b55e4b051dfb4e7e11a1ae40edbb73c396943aa63a1273485380f0daf1 WatchSource:0}: Error finding container 097858b55e4b051dfb4e7e11a1ae40edbb73c396943aa63a1273485380f0daf1: Status 404 returned error can't find the container with id 097858b55e4b051dfb4e7e11a1ae40edbb73c396943aa63a1273485380f0daf1
Jul 10 18:48:08.070988 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:08.070955    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv" event={"ID":"9637a42f-c05d-4370-9c49-a5d235015f92","Type":"ContainerStarted","Data":"330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393"}
Jul 10 18:48:08.071333 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:08.071005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv" event={"ID":"9637a42f-c05d-4370-9c49-a5d235015f92","Type":"ContainerStarted","Data":"097858b55e4b051dfb4e7e11a1ae40edbb73c396943aa63a1273485380f0daf1"}
Jul 10 18:48:08.088059 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:08.088011    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv" podStartSLOduration=1.7743488589999998 podStartE2EDuration="2.087996113s" totalImagesPullingTime="313.647254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:08.087277647 +0000 UTC m=+533.348942420" watchObservedRunningTime="2026-07-10 18:48:08.087996113 +0000 UTC m=+533.349660886"
Jul 10 18:48:09.137622 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:09.137583    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"]
Jul 10 18:48:10.076870 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:10.076800    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv" podUID="9637a42f-c05d-4370-9c49-a5d235015f92" containerName="seaweedfs-tls-custom" containerID="cri-o://330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393" gracePeriod=30
Jul 10 18:48:11.318091 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:11.318070    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:11.428610 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:11.428532    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdb9x\" (UniqueName: \"kubernetes.io/projected/9637a42f-c05d-4370-9c49-a5d235015f92-kube-api-access-tdb9x\") pod \"9637a42f-c05d-4370-9c49-a5d235015f92\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") "
Jul 10 18:48:11.428610 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:11.428581    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9637a42f-c05d-4370-9c49-a5d235015f92-data\") pod \"9637a42f-c05d-4370-9c49-a5d235015f92\" (UID: \"9637a42f-c05d-4370-9c49-a5d235015f92\") "
Jul 10 18:48:11.429919 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:11.429892    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9637a42f-c05d-4370-9c49-a5d235015f92-data" (OuterVolumeSpecName: "data") pod "9637a42f-c05d-4370-9c49-a5d235015f92" (UID: "9637a42f-c05d-4370-9c49-a5d235015f92"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:48:11.430589 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:11.430569    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9637a42f-c05d-4370-9c49-a5d235015f92-kube-api-access-tdb9x" (OuterVolumeSpecName: "kube-api-access-tdb9x") pod "9637a42f-c05d-4370-9c49-a5d235015f92" (UID: "9637a42f-c05d-4370-9c49-a5d235015f92"). InnerVolumeSpecName "kube-api-access-tdb9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:48:11.529494 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:11.529465    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdb9x\" (UniqueName: \"kubernetes.io/projected/9637a42f-c05d-4370-9c49-a5d235015f92-kube-api-access-tdb9x\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:48:11.529494 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:11.529487    2558 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9637a42f-c05d-4370-9c49-a5d235015f92-data\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:48:12.084389 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.084357    2558 generic.go:356] "Generic (PLEG): container finished" podID="9637a42f-c05d-4370-9c49-a5d235015f92" containerID="330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393" exitCode=0
Jul 10 18:48:12.084389 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.084391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv" event={"ID":"9637a42f-c05d-4370-9c49-a5d235015f92","Type":"ContainerDied","Data":"330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393"}
Jul 10 18:48:12.084596 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.084397    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"
Jul 10 18:48:12.084596 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.084430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv" event={"ID":"9637a42f-c05d-4370-9c49-a5d235015f92","Type":"ContainerDied","Data":"097858b55e4b051dfb4e7e11a1ae40edbb73c396943aa63a1273485380f0daf1"}
Jul 10 18:48:12.084596 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.084450    2558 scope.go:117] "RemoveContainer" containerID="330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393"
Jul 10 18:48:12.093324 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.093309    2558 scope.go:117] "RemoveContainer" containerID="330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393"
Jul 10 18:48:12.093557 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:48:12.093539    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393\": container with ID starting with 330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393 not found: ID does not exist" containerID="330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393"
Jul 10 18:48:12.093626 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.093566    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393"} err="failed to get container status \"330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393\": rpc error: code = NotFound desc = could not find container \"330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393\": container with ID starting with 330670ff8d2efb5d0daeec17dda8b18dd66ca03653b451e62732955ce032a393 not found: ID does not exist"
Jul 10 18:48:12.104347 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.104323    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"]
Jul 10 18:48:12.107945 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:12.107925    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-mz5nv"]
Jul 10 18:48:13.256486 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:13.256450    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9637a42f-c05d-4370-9c49-a5d235015f92" path="/var/lib/kubelet/pods/9637a42f-c05d-4370-9c49-a5d235015f92/volumes"
Jul 10 18:48:29.227612 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.227578    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"]
Jul 10 18:48:29.228007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.227828    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9637a42f-c05d-4370-9c49-a5d235015f92" containerName="seaweedfs-tls-custom"
Jul 10 18:48:29.228007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.227839    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9637a42f-c05d-4370-9c49-a5d235015f92" containerName="seaweedfs-tls-custom"
Jul 10 18:48:29.228007 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.227879    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9637a42f-c05d-4370-9c49-a5d235015f92" containerName="seaweedfs-tls-custom"
Jul 10 18:48:29.232297 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.232279    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.234615 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.234592    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 10 18:48:29.234615 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.234600    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 10 18:48:29.238920 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.238897    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"]
Jul 10 18:48:29.259448 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.259413    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/896aac54-26bf-485b-82fb-68a2797cdf7f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.259554 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.259474    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkp6h\" (UniqueName: \"kubernetes.io/projected/896aac54-26bf-485b-82fb-68a2797cdf7f-kube-api-access-fkp6h\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.259600 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.259545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/896aac54-26bf-485b-82fb-68a2797cdf7f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.360517 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.360484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkp6h\" (UniqueName: \"kubernetes.io/projected/896aac54-26bf-485b-82fb-68a2797cdf7f-kube-api-access-fkp6h\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.360678 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.360532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/896aac54-26bf-485b-82fb-68a2797cdf7f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.360678 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.360569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/896aac54-26bf-485b-82fb-68a2797cdf7f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.360933 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.360917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/896aac54-26bf-485b-82fb-68a2797cdf7f-data\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.363041 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.363015    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/896aac54-26bf-485b-82fb-68a2797cdf7f-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.369193 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.369169    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkp6h\" (UniqueName: \"kubernetes.io/projected/896aac54-26bf-485b-82fb-68a2797cdf7f-kube-api-access-fkp6h\") pod \"seaweedfs-tls-serving-67b7477c6d-s2b89\" (UID: \"896aac54-26bf-485b-82fb-68a2797cdf7f\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.541247 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.541219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"
Jul 10 18:48:29.662853 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:29.662726    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-s2b89"]
Jul 10 18:48:29.665389 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:48:29.665352    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896aac54_26bf_485b_82fb_68a2797cdf7f.slice/crio-cfa1b06ed05df54655a2ebd277aa6d92a608b8ef10b9ba6c7c51fb8da5ace723 WatchSource:0}: Error finding container cfa1b06ed05df54655a2ebd277aa6d92a608b8ef10b9ba6c7c51fb8da5ace723: Status 404 returned error can't find the container with id cfa1b06ed05df54655a2ebd277aa6d92a608b8ef10b9ba6c7c51fb8da5ace723
Jul 10 18:48:30.146674 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:30.146639    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89" event={"ID":"896aac54-26bf-485b-82fb-68a2797cdf7f","Type":"ContainerStarted","Data":"60606175c92eda9b6d8c979087d587197a69d933970a709a195315326192400a"}
Jul 10 18:48:30.146860 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:30.146686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89" event={"ID":"896aac54-26bf-485b-82fb-68a2797cdf7f","Type":"ContainerStarted","Data":"cfa1b06ed05df54655a2ebd277aa6d92a608b8ef10b9ba6c7c51fb8da5ace723"}
Jul 10 18:48:30.162491 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:30.162444    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-s2b89" podStartSLOduration=0.875908866 podStartE2EDuration="1.162428605s" totalImagesPullingTime="286.519739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:48:30.161542891 +0000 UTC m=+555.423207655" watchObservedRunningTime="2026-07-10 18:48:30.162428605 +0000 UTC m=+555.424093378"
Jul 10 18:48:46.742892 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.742861    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"]
Jul 10 18:48:46.746058 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.746041    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.749298 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.749277    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 10 18:48:46.750041 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.750024    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mtzvk\""
Jul 10 18:48:46.750134 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.750041    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 10 18:48:46.750134 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.750058    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 10 18:48:46.750134 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.750126    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 10 18:48:46.759109 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.759086    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"]
Jul 10 18:48:46.774913 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.774894    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jclr\" (UniqueName: \"kubernetes.io/projected/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kube-api-access-8jclr\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.775014 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.774928    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387b80a-cec2-4c71-a6ad-6c1629e723b2-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.775014 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.774947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387b80a-cec2-4c71-a6ad-6c1629e723b2-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.775014 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.774973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.876174 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.876145    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387b80a-cec2-4c71-a6ad-6c1629e723b2-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.876174 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.876177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387b80a-cec2-4c71-a6ad-6c1629e723b2-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.876340 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.876217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.876340 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.876280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jclr\" (UniqueName: \"kubernetes.io/projected/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kube-api-access-8jclr\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.876617 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.876593    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.876826 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.876807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387b80a-cec2-4c71-a6ad-6c1629e723b2-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.878674 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.878659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387b80a-cec2-4c71-a6ad-6c1629e723b2-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:46.892111 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:46.892089    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jclr\" (UniqueName: \"kubernetes.io/projected/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kube-api-access-8jclr\") pod \"isvc-sklearn-batcher-predictor-bbb4dd49b-pn698\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:47.057037 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:47.056972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:48:47.198375 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:47.198355    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"]
Jul 10 18:48:47.200215 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:48:47.200182    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4387b80a_cec2_4c71_a6ad_6c1629e723b2.slice/crio-afc7ad9ec41557810e9e374d70334d3ef3a0d3e62e98097f1cfc687c7f069fd0 WatchSource:0}: Error finding container afc7ad9ec41557810e9e374d70334d3ef3a0d3e62e98097f1cfc687c7f069fd0: Status 404 returned error can't find the container with id afc7ad9ec41557810e9e374d70334d3ef3a0d3e62e98097f1cfc687c7f069fd0
Jul 10 18:48:48.207678 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:48.207641    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerStarted","Data":"afc7ad9ec41557810e9e374d70334d3ef3a0d3e62e98097f1cfc687c7f069fd0"}
Jul 10 18:48:52.225054 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:52.225019    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerStarted","Data":"225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9"}
Jul 10 18:48:55.237098 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:55.237009    2558 generic.go:356] "Generic (PLEG): container finished" podID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerID="225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9" exitCode=0
Jul 10 18:48:55.237098 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:48:55.237043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerDied","Data":"225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9"}
Jul 10 18:49:10.309848 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:10.309801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerStarted","Data":"8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c"}
Jul 10 18:49:12.320329 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:12.320252    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerStarted","Data":"4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5"}
Jul 10 18:49:15.194078 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:15.193995    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:49:15.194807 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:15.194785    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:49:15.334422 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:15.334390    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerStarted","Data":"9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79"}
Jul 10 18:49:15.334574 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:15.334560    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:49:15.355152 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:15.355093    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podStartSLOduration=2.236330592 podStartE2EDuration="29.355054602s" totalImagesPullingTime="23.604043391s" totalInitContainerRuntime="3.514680619s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:48:46 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:49:15.35433426 +0000 UTC m=+600.615999033" watchObservedRunningTime="2026-07-10 18:49:15.355054602 +0000 UTC m=+600.616719376"
Jul 10 18:49:16.337939 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:16.337908    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:49:16.338296 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:16.337990    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:49:16.339194 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:16.339168    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:49:16.339780 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:16.339753    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:17.341420 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:17.341379    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:49:17.341858 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:17.341781    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:17.345093 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:17.345074    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:49:18.345018 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:18.344979    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:49:18.345495 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:18.345282    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:28.345248 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:28.345199    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:49:28.345688 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:28.345664    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:38.345911 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:38.345810    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:49:38.346327 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:38.346294    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:48.344963 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:48.344917    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:49:48.345486 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:48.345270    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:49:58.345018 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:58.344975    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:49:58.345538 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:49:58.345516    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:08.345738 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:08.345691    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:50:08.346226 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:08.346204    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:18.346350 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:18.346317    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:50:18.346842 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:18.346721    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:50:31.781517 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.781483    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"]
Jul 10 18:50:31.782397 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.781736    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" containerID="cri-o://8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c" gracePeriod=30
Jul 10 18:50:31.782397 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.781772    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" containerID="cri-o://9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79" gracePeriod=30
Jul 10 18:50:31.782397 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.781812    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy" containerID="cri-o://4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5" gracePeriod=30
Jul 10 18:50:31.791981 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.791961    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"]
Jul 10 18:50:31.794361 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.794344    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.796625 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.796605    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 10 18:50:31.796709 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.796609    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 10 18:50:31.809122 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.809098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"]
Jul 10 18:50:31.848476 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.848447    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9rb\" (UniqueName: \"kubernetes.io/projected/b8d53715-810a-4c75-b8f3-a12cead6f70e-kube-api-access-xl9rb\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.848615 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.848485    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d53715-810a-4c75-b8f3-a12cead6f70e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.848615 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.848528    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d53715-810a-4c75-b8f3-a12cead6f70e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.848615 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.848562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d53715-810a-4c75-b8f3-a12cead6f70e-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.949197 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.949171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d53715-810a-4c75-b8f3-a12cead6f70e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.949323 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.949216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d53715-810a-4c75-b8f3-a12cead6f70e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.949323 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.949246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d53715-810a-4c75-b8f3-a12cead6f70e-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.949323 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.949278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9rb\" (UniqueName: \"kubernetes.io/projected/b8d53715-810a-4c75-b8f3-a12cead6f70e-kube-api-access-xl9rb\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.949645 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.949627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d53715-810a-4c75-b8f3-a12cead6f70e-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.950005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.949987    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d53715-810a-4c75-b8f3-a12cead6f70e-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.951493 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.951474    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d53715-810a-4c75-b8f3-a12cead6f70e-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:31.958920 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:31.958900    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9rb\" (UniqueName: \"kubernetes.io/projected/b8d53715-810a-4c75-b8f3-a12cead6f70e-kube-api-access-xl9rb\") pod \"isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:32.104401 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:32.104321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:32.231313 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:32.231291    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"]
Jul 10 18:50:32.233802 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:50:32.233770    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d53715_810a_4c75_b8f3_a12cead6f70e.slice/crio-e318efa52bd62ea538e0f42d4c44a953b576902941b355c253d46550ec3e8a80 WatchSource:0}: Error finding container e318efa52bd62ea538e0f42d4c44a953b576902941b355c253d46550ec3e8a80: Status 404 returned error can't find the container with id e318efa52bd62ea538e0f42d4c44a953b576902941b355c253d46550ec3e8a80
Jul 10 18:50:32.342088 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:32.342047    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 10 18:50:32.609417 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:32.609380    2558 generic.go:356] "Generic (PLEG): container finished" podID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerID="4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5" exitCode=2
Jul 10 18:50:32.609601 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:32.609454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerDied","Data":"4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5"}
Jul 10 18:50:32.611064 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:32.611034    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerStarted","Data":"7d8e683377dbda5265e3a2b755f532f82e0a9e9512c46d9a7969eeeb4170a47e"}
Jul 10 18:50:32.611205 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:32.611087    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerStarted","Data":"e318efa52bd62ea538e0f42d4c44a953b576902941b355c253d46550ec3e8a80"}
Jul 10 18:50:36.626615 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:36.626584    2558 generic.go:356] "Generic (PLEG): container finished" podID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerID="8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c" exitCode=0
Jul 10 18:50:36.627068 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:36.626658    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerDied","Data":"8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c"}
Jul 10 18:50:36.627986 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:36.627965    2558 generic.go:356] "Generic (PLEG): container finished" podID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerID="7d8e683377dbda5265e3a2b755f532f82e0a9e9512c46d9a7969eeeb4170a47e" exitCode=0
Jul 10 18:50:36.628096 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:36.628018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerDied","Data":"7d8e683377dbda5265e3a2b755f532f82e0a9e9512c46d9a7969eeeb4170a47e"}
Jul 10 18:50:37.342155 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.342087    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 10 18:50:37.634562 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.634487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerStarted","Data":"9cec51f887636064df2da1f66198c7e011602932296641aad363af521f68f7f2"}
Jul 10 18:50:37.634954 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.634577    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerStarted","Data":"361c8be741b3d66512663b78a2ac4ad1d237f61486daf0de96e8315dcfb12238"}
Jul 10 18:50:37.634954 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.634605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerStarted","Data":"e1e3ae6eef922ce3279396f60b51b4b8c49f036664bd78a3c35dd562a4111b92"}
Jul 10 18:50:37.634954 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.634887    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:37.635109 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.635059    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:37.636320 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.636297    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:50:37.656551 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:37.656503    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podStartSLOduration=2.754408877 podStartE2EDuration="6.656489449s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.902080572s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:50:37.65511676 +0000 UTC m=+682.916781533" watchObservedRunningTime="2026-07-10 18:50:37.656489449 +0000 UTC m=+682.918154223"
Jul 10 18:50:38.345723 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:38.345688    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:38.346883 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:38.346857    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:50:38.638383 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:38.638288    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:50:38.638383 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:38.638334    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:38.639417 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:38.639389    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:39.641956 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:39.641915    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:50:39.642382 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:39.642264    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:42.341517 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:42.341477    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 10 18:50:42.341954 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:42.341597    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:50:44.646621 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:44.646591    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:50:44.647317 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:44.647274    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:50:44.647504 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:44.647479    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:47.341898 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:47.341854    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 10 18:50:48.346042 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:48.346003    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:48.347181 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:48.347154    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:50:52.342228 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:52.342178    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 10 18:50:54.647866 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:54.647824    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:50:54.648374 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:54.648218    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:57.342213 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:57.342167    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Jul 10 18:50:58.346020 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:58.345979    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:50:58.346443 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:58.346139    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:50:58.347240 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:58.347216    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 10 18:50:58.347354 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:50:58.347339    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:51:01.932964 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:01.932940    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:51:02.090037 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.089944    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jclr\" (UniqueName: \"kubernetes.io/projected/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kube-api-access-8jclr\") pod \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") "
Jul 10 18:51:02.090037 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.089984    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387b80a-cec2-4c71-a6ad-6c1629e723b2-proxy-tls\") pod \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") "
Jul 10 18:51:02.090037 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.090036    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kserve-provision-location\") pod \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") "
Jul 10 18:51:02.090256 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.090074    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387b80a-cec2-4c71-a6ad-6c1629e723b2-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\" (UID: \"4387b80a-cec2-4c71-a6ad-6c1629e723b2\") "
Jul 10 18:51:02.090447 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.090421    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4387b80a-cec2-4c71-a6ad-6c1629e723b2" (UID: "4387b80a-cec2-4c71-a6ad-6c1629e723b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:51:02.090503 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.090463    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4387b80a-cec2-4c71-a6ad-6c1629e723b2-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "4387b80a-cec2-4c71-a6ad-6c1629e723b2" (UID: "4387b80a-cec2-4c71-a6ad-6c1629e723b2"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:51:02.092280 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.092250    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4387b80a-cec2-4c71-a6ad-6c1629e723b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4387b80a-cec2-4c71-a6ad-6c1629e723b2" (UID: "4387b80a-cec2-4c71-a6ad-6c1629e723b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:51:02.092280 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.092270    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kube-api-access-8jclr" (OuterVolumeSpecName: "kube-api-access-8jclr") pod "4387b80a-cec2-4c71-a6ad-6c1629e723b2" (UID: "4387b80a-cec2-4c71-a6ad-6c1629e723b2"). InnerVolumeSpecName "kube-api-access-8jclr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:51:02.191103 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.191064    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jclr\" (UniqueName: \"kubernetes.io/projected/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kube-api-access-8jclr\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:51:02.191103 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.191095    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4387b80a-cec2-4c71-a6ad-6c1629e723b2-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:51:02.191103 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.191105    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4387b80a-cec2-4c71-a6ad-6c1629e723b2-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:51:02.191103 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.191114    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4387b80a-cec2-4c71-a6ad-6c1629e723b2-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:51:02.726623 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.726592    2558 generic.go:356] "Generic (PLEG): container finished" podID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerID="9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79" exitCode=0
Jul 10 18:51:02.726839 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.726634    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerDied","Data":"9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79"}
Jul 10 18:51:02.726839 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.726661    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"
Jul 10 18:51:02.726839 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.726689    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698" event={"ID":"4387b80a-cec2-4c71-a6ad-6c1629e723b2","Type":"ContainerDied","Data":"afc7ad9ec41557810e9e374d70334d3ef3a0d3e62e98097f1cfc687c7f069fd0"}
Jul 10 18:51:02.726839 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.726720    2558 scope.go:117] "RemoveContainer" containerID="9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79"
Jul 10 18:51:02.734726 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.734707    2558 scope.go:117] "RemoveContainer" containerID="4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5"
Jul 10 18:51:02.741600 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.741579    2558 scope.go:117] "RemoveContainer" containerID="8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c"
Jul 10 18:51:02.748769 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.748732    2558 scope.go:117] "RemoveContainer" containerID="225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9"
Jul 10 18:51:02.752364 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.752339    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"]
Jul 10 18:51:02.755587 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.755566    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-bbb4dd49b-pn698"]
Jul 10 18:51:02.756434 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.756421    2558 scope.go:117] "RemoveContainer" containerID="9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79"
Jul 10 18:51:02.756719 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:51:02.756703    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79\": container with ID starting with 9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79 not found: ID does not exist" containerID="9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79"
Jul 10 18:51:02.756787 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.756731    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79"} err="failed to get container status \"9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79\": rpc error: code = NotFound desc = could not find container \"9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79\": container with ID starting with 9dbd675b64fceb5a42ce213ab2044b4caa23557933ab2acb1b7086c137bb7e79 not found: ID does not exist"
Jul 10 18:51:02.756787 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.756768    2558 scope.go:117] "RemoveContainer" containerID="4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5"
Jul 10 18:51:02.757043 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:51:02.757024    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5\": container with ID starting with 4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5 not found: ID does not exist" containerID="4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5"
Jul 10 18:51:02.757102 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.757046    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5"} err="failed to get container status \"4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5\": rpc error: code = NotFound desc = could not find container \"4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5\": container with ID starting with 4247bdc7d113f433ef5118d547822b83e75bbf42d9dca690da3c9d9c9aa870f5 not found: ID does not exist"
Jul 10 18:51:02.757102 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.757066    2558 scope.go:117] "RemoveContainer" containerID="8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c"
Jul 10 18:51:02.757299 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:51:02.757282    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c\": container with ID starting with 8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c not found: ID does not exist" containerID="8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c"
Jul 10 18:51:02.757337 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.757304    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c"} err="failed to get container status \"8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c\": rpc error: code = NotFound desc = could not find container \"8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c\": container with ID starting with 8519ee586e6dc369dadd4f286b9f9a9d8ac915ba28f54d7eae0ed7508e837e9c not found: ID does not exist"
Jul 10 18:51:02.757337 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.757321    2558 scope.go:117] "RemoveContainer" containerID="225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9"
Jul 10 18:51:02.757551 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:51:02.757535    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9\": container with ID starting with 225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9 not found: ID does not exist" containerID="225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9"
Jul 10 18:51:02.757600 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:02.757555    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9"} err="failed to get container status \"225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9\": rpc error: code = NotFound desc = could not find container \"225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9\": container with ID starting with 225eccf6b062c7aecf84da93da44df44f3647b3a66c42c64f1ef271869626de9 not found: ID does not exist"
Jul 10 18:51:03.256939 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:03.256910    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" path="/var/lib/kubelet/pods/4387b80a-cec2-4c71-a6ad-6c1629e723b2/volumes"
Jul 10 18:51:04.647695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:04.647652    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:51:04.648198 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:04.648105    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:14.647871 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:14.647776    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:51:14.648361 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:14.648125    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:24.647120 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:24.647078    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:51:24.647638 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:24.647607    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:34.647786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:34.647729    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:51:34.648233 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:34.648180    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:51:44.648619 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:44.648588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:51:44.649065 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:44.648759    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:51:56.916940 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.916906    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"]
Jul 10 18:51:56.917383 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917319    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="storage-initializer"
Jul 10 18:51:56.917383 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917336    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="storage-initializer"
Jul 10 18:51:56.917383 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917352    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container"
Jul 10 18:51:56.917383 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917361    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container"
Jul 10 18:51:56.917383 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917384    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent"
Jul 10 18:51:56.917629 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917393    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent"
Jul 10 18:51:56.917629 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917409    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy"
Jul 10 18:51:56.917629 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917418    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy"
Jul 10 18:51:56.917629 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917490    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="agent"
Jul 10 18:51:56.917629 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917505    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kserve-container"
Jul 10 18:51:56.917629 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.917514    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4387b80a-cec2-4c71-a6ad-6c1629e723b2" containerName="kube-rbac-proxy"
Jul 10 18:51:56.920508 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.920485    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:56.922948 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.922923    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 10 18:51:56.924012 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.923982    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 10 18:51:56.925885 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.925862    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"]
Jul 10 18:51:56.926237 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.926210    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" containerID="cri-o://e1e3ae6eef922ce3279396f60b51b4b8c49f036664bd78a3c35dd562a4111b92" gracePeriod=30
Jul 10 18:51:56.926464 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.926249    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" containerID="cri-o://9cec51f887636064df2da1f66198c7e011602932296641aad363af521f68f7f2" gracePeriod=30
Jul 10 18:51:56.926563 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.926261    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy" containerID="cri-o://361c8be741b3d66512663b78a2ac4ad1d237f61486daf0de96e8315dcfb12238" gracePeriod=30
Jul 10 18:51:56.932241 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.932209    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"]
Jul 10 18:51:56.976367 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.976335    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63753769-9dd1-4e27-9683-e53c43fc12af-proxy-tls\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:56.976540 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.976378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63753769-9dd1-4e27-9683-e53c43fc12af-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:56.976540 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:56.976396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfrr\" (UniqueName: \"kubernetes.io/projected/63753769-9dd1-4e27-9683-e53c43fc12af-kube-api-access-7dfrr\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.077429 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.077392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63753769-9dd1-4e27-9683-e53c43fc12af-proxy-tls\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.077589 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.077435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63753769-9dd1-4e27-9683-e53c43fc12af-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.077589 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.077462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfrr\" (UniqueName: \"kubernetes.io/projected/63753769-9dd1-4e27-9683-e53c43fc12af-kube-api-access-7dfrr\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.078194 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.078171    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63753769-9dd1-4e27-9683-e53c43fc12af-message-dumper-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.079771 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.079738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63753769-9dd1-4e27-9683-e53c43fc12af-proxy-tls\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.085096 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.085077    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfrr\" (UniqueName: \"kubernetes.io/projected/63753769-9dd1-4e27-9683-e53c43fc12af-kube-api-access-7dfrr\") pod \"message-dumper-predictor-5d6f5c7c4b-69qpm\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") " pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.234437 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.234349    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:57.368508 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.368479    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"]
Jul 10 18:51:57.371520 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:51:57.371479    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63753769_9dd1_4e27_9683_e53c43fc12af.slice/crio-e00126f0a90d07f50b48401e37bf9ad4a3ca38ae0ef5f9864014861ef0b04d7f WatchSource:0}: Error finding container e00126f0a90d07f50b48401e37bf9ad4a3ca38ae0ef5f9864014861ef0b04d7f: Status 404 returned error can't find the container with id e00126f0a90d07f50b48401e37bf9ad4a3ca38ae0ef5f9864014861ef0b04d7f
Jul 10 18:51:57.373281 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.373267    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 18:51:57.918659 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.918626    2558 generic.go:356] "Generic (PLEG): container finished" podID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerID="361c8be741b3d66512663b78a2ac4ad1d237f61486daf0de96e8315dcfb12238" exitCode=2
Jul 10 18:51:57.919055 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.918693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerDied","Data":"361c8be741b3d66512663b78a2ac4ad1d237f61486daf0de96e8315dcfb12238"}
Jul 10 18:51:57.919859 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:57.919832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" event={"ID":"63753769-9dd1-4e27-9683-e53c43fc12af","Type":"ContainerStarted","Data":"e00126f0a90d07f50b48401e37bf9ad4a3ca38ae0ef5f9864014861ef0b04d7f"}
Jul 10 18:51:58.925144 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:58.925105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" event={"ID":"63753769-9dd1-4e27-9683-e53c43fc12af","Type":"ContainerStarted","Data":"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"}
Jul 10 18:51:58.925520 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:58.925160    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" event={"ID":"63753769-9dd1-4e27-9683-e53c43fc12af","Type":"ContainerStarted","Data":"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"}
Jul 10 18:51:58.925520 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:58.925335    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:58.925520 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:58.925443    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:58.927107 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:58.927085    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:51:58.945269 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:58.945225    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" podStartSLOduration=1.573158335 podStartE2EDuration="2.945209559s" totalImagesPullingTime="1.372051224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:51:58.943715133 +0000 UTC m=+764.205379903" watchObservedRunningTime="2026-07-10 18:51:58.945209559 +0000 UTC m=+764.206874332"
Jul 10 18:51:59.642345 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:51:59.642303    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 10 18:52:01.944385 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:01.944348    2558 generic.go:356] "Generic (PLEG): container finished" podID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerID="e1e3ae6eef922ce3279396f60b51b4b8c49f036664bd78a3c35dd562a4111b92" exitCode=0
Jul 10 18:52:01.944768 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:01.944402    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerDied","Data":"e1e3ae6eef922ce3279396f60b51b4b8c49f036664bd78a3c35dd562a4111b92"}
Jul 10 18:52:04.642207 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:04.642163    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 10 18:52:04.647509 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:04.647479    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:52:04.647803 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:04.647781    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:05.943260 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:05.943232    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:52:06.980895 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:06.980862    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"]
Jul 10 18:52:06.984138 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:06.984120    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:06.986263 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:06.986243    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 10 18:52:06.986521 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:06.986503    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 10 18:52:06.998964 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:06.998934    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"]
Jul 10 18:52:07.052910 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.052873    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvqc\" (UniqueName: \"kubernetes.io/projected/4b128ead-e6ae-471e-be5e-0df7df6150ee-kube-api-access-9zvqc\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.053083 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.052932    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.053083 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.052952    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b128ead-e6ae-471e-be5e-0df7df6150ee-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.053083 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.053007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b128ead-e6ae-471e-be5e-0df7df6150ee-kserve-provision-location\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.153567 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.153524    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvqc\" (UniqueName: \"kubernetes.io/projected/4b128ead-e6ae-471e-be5e-0df7df6150ee-kube-api-access-9zvqc\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.153810 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.153589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.153810 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.153613    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b128ead-e6ae-471e-be5e-0df7df6150ee-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.153810 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.153659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b128ead-e6ae-471e-be5e-0df7df6150ee-kserve-provision-location\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.153810 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:52:07.153731    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 10 18:52:07.154010 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:52:07.153853    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls podName:4b128ead-e6ae-471e-be5e-0df7df6150ee nodeName:}" failed. No retries permitted until 2026-07-10 18:52:07.653823296 +0000 UTC m=+772.915488060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls") pod "isvc-logger-predictor-78b64bf47-blvd2" (UID: "4b128ead-e6ae-471e-be5e-0df7df6150ee") : secret "isvc-logger-predictor-serving-cert" not found
Jul 10 18:52:07.154068 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.154040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b128ead-e6ae-471e-be5e-0df7df6150ee-kserve-provision-location\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.154411 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.154391    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b128ead-e6ae-471e-be5e-0df7df6150ee-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.164788 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.164767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvqc\" (UniqueName: \"kubernetes.io/projected/4b128ead-e6ae-471e-be5e-0df7df6150ee-kube-api-access-9zvqc\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.657554 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.657519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.660034 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.660009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls\") pod \"isvc-logger-predictor-78b64bf47-blvd2\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:07.894614 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:07.894578    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:08.023856 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:08.023781    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"]
Jul 10 18:52:08.026440 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:52:08.026413    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b128ead_e6ae_471e_be5e_0df7df6150ee.slice/crio-0d9e6f77f9e8c5e040bc8dba3a50fff86d437c20b8c9f829ebdfa93cbd66ce0d WatchSource:0}: Error finding container 0d9e6f77f9e8c5e040bc8dba3a50fff86d437c20b8c9f829ebdfa93cbd66ce0d: Status 404 returned error can't find the container with id 0d9e6f77f9e8c5e040bc8dba3a50fff86d437c20b8c9f829ebdfa93cbd66ce0d
Jul 10 18:52:08.971689 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:08.971657    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerStarted","Data":"2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc"}
Jul 10 18:52:08.971851 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:08.971723    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerStarted","Data":"0d9e6f77f9e8c5e040bc8dba3a50fff86d437c20b8c9f829ebdfa93cbd66ce0d"}
Jul 10 18:52:09.642879 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:09.642833    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 10 18:52:09.643286 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:09.642977    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:52:11.984998 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:11.984965    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerID="2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc" exitCode=0
Jul 10 18:52:11.985365 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:11.985038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerDied","Data":"2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc"}
Jul 10 18:52:12.996352 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.996310    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerStarted","Data":"28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63"}
Jul 10 18:52:12.997251 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.996433    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerStarted","Data":"18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627"}
Jul 10 18:52:12.997251 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.996469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerStarted","Data":"6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6"}
Jul 10 18:52:12.997251 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.996850    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:12.997251 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.996916    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:12.997251 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.996951    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:12.998483 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.998427    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:52:12.999187 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:12.999156    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:13.019475 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:13.019433    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podStartSLOduration=3.309624572 podStartE2EDuration="7.019419648s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.709795076s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:52:13.018230051 +0000 UTC m=+778.279894848" watchObservedRunningTime="2026-07-10 18:52:13.019419648 +0000 UTC m=+778.281084466"
Jul 10 18:52:14.001318 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:14.001277    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:52:14.001786 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:14.001611    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:14.642585 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:14.642537    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 10 18:52:14.647999 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:14.647966    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:52:14.648329 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:14.648307    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:19.006349 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:19.006317    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:52:19.006981 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:19.006947    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:52:19.007407 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:19.007383    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:19.642891 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:19.642849    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 10 18:52:24.642791 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:24.642730    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Jul 10 18:52:24.647124 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:24.647095    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Jul 10 18:52:24.647265 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:24.647251    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:52:24.647387 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:24.647363    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:24.647483 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:24.647469    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:52:27.051620 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.051590    2558 generic.go:356] "Generic (PLEG): container finished" podID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerID="9cec51f887636064df2da1f66198c7e011602932296641aad363af521f68f7f2" exitCode=0
Jul 10 18:52:27.051987 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.051644    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerDied","Data":"9cec51f887636064df2da1f66198c7e011602932296641aad363af521f68f7f2"}
Jul 10 18:52:27.076529 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.076508    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:52:27.213284 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.213194    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl9rb\" (UniqueName: \"kubernetes.io/projected/b8d53715-810a-4c75-b8f3-a12cead6f70e-kube-api-access-xl9rb\") pod \"b8d53715-810a-4c75-b8f3-a12cead6f70e\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") "
Jul 10 18:52:27.213284 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.213277    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d53715-810a-4c75-b8f3-a12cead6f70e-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"b8d53715-810a-4c75-b8f3-a12cead6f70e\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") "
Jul 10 18:52:27.213487 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.213405    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d53715-810a-4c75-b8f3-a12cead6f70e-kserve-provision-location\") pod \"b8d53715-810a-4c75-b8f3-a12cead6f70e\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") "
Jul 10 18:52:27.213487 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.213449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d53715-810a-4c75-b8f3-a12cead6f70e-proxy-tls\") pod \"b8d53715-810a-4c75-b8f3-a12cead6f70e\" (UID: \"b8d53715-810a-4c75-b8f3-a12cead6f70e\") "
Jul 10 18:52:27.213592 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.213566    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d53715-810a-4c75-b8f3-a12cead6f70e-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "b8d53715-810a-4c75-b8f3-a12cead6f70e" (UID: "b8d53715-810a-4c75-b8f3-a12cead6f70e"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:52:27.213693 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.213679    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d53715-810a-4c75-b8f3-a12cead6f70e-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:52:27.213737 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.213682    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d53715-810a-4c75-b8f3-a12cead6f70e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8d53715-810a-4c75-b8f3-a12cead6f70e" (UID: "b8d53715-810a-4c75-b8f3-a12cead6f70e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:52:27.215384 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.215354    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d53715-810a-4c75-b8f3-a12cead6f70e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8d53715-810a-4c75-b8f3-a12cead6f70e" (UID: "b8d53715-810a-4c75-b8f3-a12cead6f70e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:52:27.215498 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.215405    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d53715-810a-4c75-b8f3-a12cead6f70e-kube-api-access-xl9rb" (OuterVolumeSpecName: "kube-api-access-xl9rb") pod "b8d53715-810a-4c75-b8f3-a12cead6f70e" (UID: "b8d53715-810a-4c75-b8f3-a12cead6f70e"). InnerVolumeSpecName "kube-api-access-xl9rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:52:27.314981 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.314952    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9rb\" (UniqueName: \"kubernetes.io/projected/b8d53715-810a-4c75-b8f3-a12cead6f70e-kube-api-access-xl9rb\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:52:27.314981 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.314975    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d53715-810a-4c75-b8f3-a12cead6f70e-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:52:27.314981 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:27.314985    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d53715-810a-4c75-b8f3-a12cead6f70e-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:52:28.057479 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.057447    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5" event={"ID":"b8d53715-810a-4c75-b8f3-a12cead6f70e","Type":"ContainerDied","Data":"e318efa52bd62ea538e0f42d4c44a953b576902941b355c253d46550ec3e8a80"}
Jul 10 18:52:28.057479 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.057470    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"
Jul 10 18:52:28.057953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.057519    2558 scope.go:117] "RemoveContainer" containerID="9cec51f887636064df2da1f66198c7e011602932296641aad363af521f68f7f2"
Jul 10 18:52:28.065121 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.065104    2558 scope.go:117] "RemoveContainer" containerID="361c8be741b3d66512663b78a2ac4ad1d237f61486daf0de96e8315dcfb12238"
Jul 10 18:52:28.072349 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.072332    2558 scope.go:117] "RemoveContainer" containerID="e1e3ae6eef922ce3279396f60b51b4b8c49f036664bd78a3c35dd562a4111b92"
Jul 10 18:52:28.075153 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.075132    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"]
Jul 10 18:52:28.080148 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.080124    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5548f66465-tmwk5"]
Jul 10 18:52:28.080681 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:28.080654    2558 scope.go:117] "RemoveContainer" containerID="7d8e683377dbda5265e3a2b755f532f82e0a9e9512c46d9a7969eeeb4170a47e"
Jul 10 18:52:29.007844 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:29.007804    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:52:29.008408 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:29.008376    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:29.257260 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:29.257225    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" path="/var/lib/kubelet/pods/b8d53715-810a-4c75-b8f3-a12cead6f70e/volumes"
Jul 10 18:52:39.007861 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:39.007768    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:52:39.008368 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:39.008171    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:49.007605 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:49.007558    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:52:49.008048 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:49.008009    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:52:59.007439 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:59.007403    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:52:59.007949 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:52:59.007926    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:09.007005 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:09.006959    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:53:09.007473 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:09.007440    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:19.007628 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:19.007588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:53:19.008051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:19.007796    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:53:32.003377 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.003345    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-5d6f5c7c4b-69qpm_63753769-9dd1-4e27-9683-e53c43fc12af/kserve-container/0.log"
Jul 10 18:53:32.190388 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.190347    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"]
Jul 10 18:53:32.191881 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.191827    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" containerID="cri-o://28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63" gracePeriod=30
Jul 10 18:53:32.192201 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.192181    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy" containerID="cri-o://18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627" gracePeriod=30
Jul 10 18:53:32.192321 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.192286    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" containerID="cri-o://6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6" gracePeriod=30
Jul 10 18:53:32.229138 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229114    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"]
Jul 10 18:53:32.229378 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229367    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent"
Jul 10 18:53:32.229419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229379    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent"
Jul 10 18:53:32.229419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229393    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container"
Jul 10 18:53:32.229419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229398    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container"
Jul 10 18:53:32.229419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229405    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy"
Jul 10 18:53:32.229419 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229410    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy"
Jul 10 18:53:32.229575 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229424    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="storage-initializer"
Jul 10 18:53:32.229575 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229430    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="storage-initializer"
Jul 10 18:53:32.229575 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229471    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kube-rbac-proxy"
Jul 10 18:53:32.229575 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229482    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="kserve-container"
Jul 10 18:53:32.229575 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.229487    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d53715-810a-4c75-b8f3-a12cead6f70e" containerName="agent"
Jul 10 18:53:32.233427 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.233410    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.235919 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.235898    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 10 18:53:32.236055 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.235900    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 10 18:53:32.242727 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.242689    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"]
Jul 10 18:53:32.292470 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.292385    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bb1fae-d502-4da3-96cb-25de539d8a82-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.292470 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.292435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bb1fae-d502-4da3-96cb-25de539d8a82-kserve-provision-location\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.292652 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.292505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bb1fae-d502-4da3-96cb-25de539d8a82-proxy-tls\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.292652 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.292554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8ht\" (UniqueName: \"kubernetes.io/projected/48bb1fae-d502-4da3-96cb-25de539d8a82-kube-api-access-br8ht\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.308818 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.308789    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"]
Jul 10 18:53:32.309032 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.309009    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kserve-container" containerID="cri-o://25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5" gracePeriod=30
Jul 10 18:53:32.309075 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.309035    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kube-rbac-proxy" containerID="cri-o://3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27" gracePeriod=30
Jul 10 18:53:32.393652 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.393619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8ht\" (UniqueName: \"kubernetes.io/projected/48bb1fae-d502-4da3-96cb-25de539d8a82-kube-api-access-br8ht\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.393828 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.393668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bb1fae-d502-4da3-96cb-25de539d8a82-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.393828 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.393692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bb1fae-d502-4da3-96cb-25de539d8a82-kserve-provision-location\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.393828 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.393720    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bb1fae-d502-4da3-96cb-25de539d8a82-proxy-tls\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.394144 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.394127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bb1fae-d502-4da3-96cb-25de539d8a82-kserve-provision-location\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.394410 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.394392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bb1fae-d502-4da3-96cb-25de539d8a82-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.396046 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.396030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bb1fae-d502-4da3-96cb-25de539d8a82-proxy-tls\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.402790 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.402770    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8ht\" (UniqueName: \"kubernetes.io/projected/48bb1fae-d502-4da3-96cb-25de539d8a82-kube-api-access-br8ht\") pod \"isvc-lightgbm-predictor-57dd68dd5d-nbhqz\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.544630 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.544566    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:53:32.544630 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.544601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:32.595280 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.595246    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63753769-9dd1-4e27-9683-e53c43fc12af-message-dumper-kube-rbac-proxy-sar-config\") pod \"63753769-9dd1-4e27-9683-e53c43fc12af\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") "
Jul 10 18:53:32.595399 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.595326    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63753769-9dd1-4e27-9683-e53c43fc12af-proxy-tls\") pod \"63753769-9dd1-4e27-9683-e53c43fc12af\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") "
Jul 10 18:53:32.595470 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.595413    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dfrr\" (UniqueName: \"kubernetes.io/projected/63753769-9dd1-4e27-9683-e53c43fc12af-kube-api-access-7dfrr\") pod \"63753769-9dd1-4e27-9683-e53c43fc12af\" (UID: \"63753769-9dd1-4e27-9683-e53c43fc12af\") "
Jul 10 18:53:32.595727 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.595680    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63753769-9dd1-4e27-9683-e53c43fc12af-message-dumper-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-kube-rbac-proxy-sar-config") pod "63753769-9dd1-4e27-9683-e53c43fc12af" (UID: "63753769-9dd1-4e27-9683-e53c43fc12af"). InnerVolumeSpecName "message-dumper-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:53:32.597432 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.597405    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63753769-9dd1-4e27-9683-e53c43fc12af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63753769-9dd1-4e27-9683-e53c43fc12af" (UID: "63753769-9dd1-4e27-9683-e53c43fc12af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:53:32.597676 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.597651    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63753769-9dd1-4e27-9683-e53c43fc12af-kube-api-access-7dfrr" (OuterVolumeSpecName: "kube-api-access-7dfrr") pod "63753769-9dd1-4e27-9683-e53c43fc12af" (UID: "63753769-9dd1-4e27-9683-e53c43fc12af"). InnerVolumeSpecName "kube-api-access-7dfrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:53:32.674033 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.674013    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"]
Jul 10 18:53:32.676597 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:53:32.676566    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bb1fae_d502_4da3_96cb_25de539d8a82.slice/crio-d321773a6aa36e21aaec3bb92287d78f73569fed377ddfe35fa16a5213d93ba1 WatchSource:0}: Error finding container d321773a6aa36e21aaec3bb92287d78f73569fed377ddfe35fa16a5213d93ba1: Status 404 returned error can't find the container with id d321773a6aa36e21aaec3bb92287d78f73569fed377ddfe35fa16a5213d93ba1
Jul 10 18:53:32.695897 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.695876    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfrr\" (UniqueName: \"kubernetes.io/projected/63753769-9dd1-4e27-9683-e53c43fc12af-kube-api-access-7dfrr\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:53:32.695897 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.695896    2558 reconciler_common.go:299] "Volume detached for volume \"message-dumper-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63753769-9dd1-4e27-9683-e53c43fc12af-message-dumper-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:53:32.696013 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:32.695906    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63753769-9dd1-4e27-9683-e53c43fc12af-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:53:33.301027 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.300999    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerID="18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627" exitCode=2
Jul 10 18:53:33.301434 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.301072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerDied","Data":"18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627"}
Jul 10 18:53:33.302602 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.302574    2558 generic.go:356] "Generic (PLEG): container finished" podID="63753769-9dd1-4e27-9683-e53c43fc12af" containerID="3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27" exitCode=2
Jul 10 18:53:33.302602 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.302602    2558 generic.go:356] "Generic (PLEG): container finished" podID="63753769-9dd1-4e27-9683-e53c43fc12af" containerID="25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5" exitCode=2
Jul 10 18:53:33.302785 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.302609    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" event={"ID":"63753769-9dd1-4e27-9683-e53c43fc12af","Type":"ContainerDied","Data":"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"}
Jul 10 18:53:33.302785 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.302622    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"
Jul 10 18:53:33.302785 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.302658    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" event={"ID":"63753769-9dd1-4e27-9683-e53c43fc12af","Type":"ContainerDied","Data":"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"}
Jul 10 18:53:33.302785 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.302679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm" event={"ID":"63753769-9dd1-4e27-9683-e53c43fc12af","Type":"ContainerDied","Data":"e00126f0a90d07f50b48401e37bf9ad4a3ca38ae0ef5f9864014861ef0b04d7f"}
Jul 10 18:53:33.302785 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.302704    2558 scope.go:117] "RemoveContainer" containerID="3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"
Jul 10 18:53:33.304337 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.304311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerStarted","Data":"be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5"}
Jul 10 18:53:33.304436 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.304367    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerStarted","Data":"d321773a6aa36e21aaec3bb92287d78f73569fed377ddfe35fa16a5213d93ba1"}
Jul 10 18:53:33.311342 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.311328    2558 scope.go:117] "RemoveContainer" containerID="25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"
Jul 10 18:53:33.318222 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.318205    2558 scope.go:117] "RemoveContainer" containerID="3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"
Jul 10 18:53:33.318468 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:53:33.318450    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27\": container with ID starting with 3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27 not found: ID does not exist" containerID="3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"
Jul 10 18:53:33.318523 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.318476    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"} err="failed to get container status \"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27\": rpc error: code = NotFound desc = could not find container \"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27\": container with ID starting with 3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27 not found: ID does not exist"
Jul 10 18:53:33.318523 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.318493    2558 scope.go:117] "RemoveContainer" containerID="25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"
Jul 10 18:53:33.318696 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:53:33.318680    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5\": container with ID starting with 25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5 not found: ID does not exist" containerID="25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"
Jul 10 18:53:33.318757 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.318698    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"} err="failed to get container status \"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5\": rpc error: code = NotFound desc = could not find container \"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5\": container with ID starting with 25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5 not found: ID does not exist"
Jul 10 18:53:33.318757 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.318710    2558 scope.go:117] "RemoveContainer" containerID="3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"
Jul 10 18:53:33.318964 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.318949    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27"} err="failed to get container status \"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27\": rpc error: code = NotFound desc = could not find container \"3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27\": container with ID starting with 3616b938ece61f81f72eaa4123c793f8c5be0a0853020b906498f33e45d87c27 not found: ID does not exist"
Jul 10 18:53:33.319013 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.318964    2558 scope.go:117] "RemoveContainer" containerID="25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"
Jul 10 18:53:33.319190 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.319170    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5"} err="failed to get container status \"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5\": rpc error: code = NotFound desc = could not find container \"25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5\": container with ID starting with 25c8e3411e6128413b7567a53824685757599ea13fcb0b9e0a001ec16e8e35d5 not found: ID does not exist"
Jul 10 18:53:33.343309 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.343286    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"]
Jul 10 18:53:33.349548 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:33.349526    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-5d6f5c7c4b-69qpm"]
Jul 10 18:53:34.002062 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:34.002011    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 10 18:53:35.257602 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:35.257570    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" path="/var/lib/kubelet/pods/63753769-9dd1-4e27-9683-e53c43fc12af/volumes"
Jul 10 18:53:36.318691 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:36.318665    2558 generic.go:356] "Generic (PLEG): container finished" podID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerID="be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5" exitCode=0
Jul 10 18:53:36.319015 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:36.318736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerDied","Data":"be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5"}
Jul 10 18:53:36.321371 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:36.321346    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerID="6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6" exitCode=0
Jul 10 18:53:36.321459 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:36.321382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerDied","Data":"6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6"}
Jul 10 18:53:39.002202 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:39.002157    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 10 18:53:39.007648 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:39.007612    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:53:39.008000 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:39.007972    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:43.354610 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:43.354575    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerStarted","Data":"905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6"}
Jul 10 18:53:43.354983 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:43.354636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerStarted","Data":"a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5"}
Jul 10 18:53:43.354983 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:43.354855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:43.375636 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:43.375571    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podStartSLOduration=1.553908811 podStartE2EDuration="11.375558513s" totalImagesPullingTime="6.290387869s" totalInitContainerRuntime="3.531261833s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:53:43.374530023 +0000 UTC m=+868.636194793" watchObservedRunningTime="2026-07-10 18:53:43.375558513 +0000 UTC m=+868.637223286"
Jul 10 18:53:44.001676 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:44.001632    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 10 18:53:44.001862 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:44.001834    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:53:44.358838 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:44.358736    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:44.360049 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:44.360023    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:53:45.362591 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:45.362553    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:53:49.002361 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:49.002305    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 10 18:53:49.007776 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:49.007719    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:53:49.008025 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:49.007994    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:50.366951 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:50.366924    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:53:50.367399 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:50.367377    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:53:54.002124 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:54.002086    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 10 18:53:59.001456 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:59.001417    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 10 18:53:59.006879 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:59.006839    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 10 18:53:59.006994 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:59.006982    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:53:59.007181 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:59.007147    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 18:53:59.007320 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:53:59.007256    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:54:00.367926 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:00.367880    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:54:02.344829 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.344791    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:54:02.422965 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.422934    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerID="28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63" exitCode=0
Jul 10 18:54:02.423135 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.422990    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"
Jul 10 18:54:02.423135 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.423016    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerDied","Data":"28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63"}
Jul 10 18:54:02.423135 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.423078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2" event={"ID":"4b128ead-e6ae-471e-be5e-0df7df6150ee","Type":"ContainerDied","Data":"0d9e6f77f9e8c5e040bc8dba3a50fff86d437c20b8c9f829ebdfa93cbd66ce0d"}
Jul 10 18:54:02.423135 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.423116    2558 scope.go:117] "RemoveContainer" containerID="28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63"
Jul 10 18:54:02.430644 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.430618    2558 scope.go:117] "RemoveContainer" containerID="18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627"
Jul 10 18:54:02.437256 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.437239    2558 scope.go:117] "RemoveContainer" containerID="6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6"
Jul 10 18:54:02.443695 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.443677    2558 scope.go:117] "RemoveContainer" containerID="2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc"
Jul 10 18:54:02.450160 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.450144    2558 scope.go:117] "RemoveContainer" containerID="28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63"
Jul 10 18:54:02.450386 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:54:02.450371    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63\": container with ID starting with 28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63 not found: ID does not exist" containerID="28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63"
Jul 10 18:54:02.450431 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.450396    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63"} err="failed to get container status \"28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63\": rpc error: code = NotFound desc = could not find container \"28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63\": container with ID starting with 28929af226ec5dadf0f9cde0a414924e86d3180a0401c73c0ef68987490b4f63 not found: ID does not exist"
Jul 10 18:54:02.450431 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.450414    2558 scope.go:117] "RemoveContainer" containerID="18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627"
Jul 10 18:54:02.450634 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:54:02.450616    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627\": container with ID starting with 18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627 not found: ID does not exist" containerID="18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627"
Jul 10 18:54:02.450682 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.450638    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627"} err="failed to get container status \"18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627\": rpc error: code = NotFound desc = could not find container \"18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627\": container with ID starting with 18404c2c169000a39e57d4550a7d6f58f1dc36f94c5943cb8660cf39ad6a8627 not found: ID does not exist"
Jul 10 18:54:02.450682 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.450654    2558 scope.go:117] "RemoveContainer" containerID="6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6"
Jul 10 18:54:02.450916 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:54:02.450896    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6\": container with ID starting with 6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6 not found: ID does not exist" containerID="6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6"
Jul 10 18:54:02.450993 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.450918    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6"} err="failed to get container status \"6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6\": rpc error: code = NotFound desc = could not find container \"6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6\": container with ID starting with 6929931e8f27b9d417d1c75accd2f5becb5f246bc120dbbd86fa72a057aabbb6 not found: ID does not exist"
Jul 10 18:54:02.450993 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.450932    2558 scope.go:117] "RemoveContainer" containerID="2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc"
Jul 10 18:54:02.451147 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:54:02.451130    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc\": container with ID starting with 2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc not found: ID does not exist" containerID="2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc"
Jul 10 18:54:02.451186 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.451150    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc"} err="failed to get container status \"2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc\": rpc error: code = NotFound desc = could not find container \"2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc\": container with ID starting with 2f27185893d671f70c20552f9b1553bc01762654c63b8d96e61cdb5c83a2bacc not found: ID does not exist"
Jul 10 18:54:02.529939 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.529911    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zvqc\" (UniqueName: \"kubernetes.io/projected/4b128ead-e6ae-471e-be5e-0df7df6150ee-kube-api-access-9zvqc\") pod \"4b128ead-e6ae-471e-be5e-0df7df6150ee\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") "
Jul 10 18:54:02.529939 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.529940    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b128ead-e6ae-471e-be5e-0df7df6150ee-kserve-provision-location\") pod \"4b128ead-e6ae-471e-be5e-0df7df6150ee\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") "
Jul 10 18:54:02.530158 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.529965    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b128ead-e6ae-471e-be5e-0df7df6150ee-isvc-logger-kube-rbac-proxy-sar-config\") pod \"4b128ead-e6ae-471e-be5e-0df7df6150ee\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") "
Jul 10 18:54:02.530158 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.530006    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls\") pod \"4b128ead-e6ae-471e-be5e-0df7df6150ee\" (UID: \"4b128ead-e6ae-471e-be5e-0df7df6150ee\") "
Jul 10 18:54:02.530281 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.530259    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b128ead-e6ae-471e-be5e-0df7df6150ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b128ead-e6ae-471e-be5e-0df7df6150ee" (UID: "4b128ead-e6ae-471e-be5e-0df7df6150ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:54:02.530345 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.530282    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b128ead-e6ae-471e-be5e-0df7df6150ee-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "4b128ead-e6ae-471e-be5e-0df7df6150ee" (UID: "4b128ead-e6ae-471e-be5e-0df7df6150ee"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:54:02.531959 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.531935    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b128ead-e6ae-471e-be5e-0df7df6150ee" (UID: "4b128ead-e6ae-471e-be5e-0df7df6150ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:54:02.532051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.532017    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b128ead-e6ae-471e-be5e-0df7df6150ee-kube-api-access-9zvqc" (OuterVolumeSpecName: "kube-api-access-9zvqc") pod "4b128ead-e6ae-471e-be5e-0df7df6150ee" (UID: "4b128ead-e6ae-471e-be5e-0df7df6150ee"). InnerVolumeSpecName "kube-api-access-9zvqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:54:02.631542 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.631503    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b128ead-e6ae-471e-be5e-0df7df6150ee-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:54:02.631542 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.631531    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b128ead-e6ae-471e-be5e-0df7df6150ee-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:54:02.631542 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.631541    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvqc\" (UniqueName: \"kubernetes.io/projected/4b128ead-e6ae-471e-be5e-0df7df6150ee-kube-api-access-9zvqc\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:54:02.631542 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.631550    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b128ead-e6ae-471e-be5e-0df7df6150ee-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:54:02.744360 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.744331    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"]
Jul 10 18:54:02.748304 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:02.748281    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-78b64bf47-blvd2"]
Jul 10 18:54:03.256186 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:03.256157    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" path="/var/lib/kubelet/pods/4b128ead-e6ae-471e-be5e-0df7df6150ee/volumes"
Jul 10 18:54:10.368446 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:10.368355    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:54:15.214979 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:15.214948    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:54:15.216527 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:15.216507    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:54:20.367381 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:20.367340    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:54:30.368006 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:30.367970    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:54:40.367969 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:40.367924    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:54:50.367505 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:54:50.367465    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 10 18:55:00.368436 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:00.368408    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:55:02.367626 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.367593    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"]
Jul 10 18:55:02.368037 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.367883    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container" containerID="cri-o://a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5" gracePeriod=30
Jul 10 18:55:02.368037 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.367956    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kube-rbac-proxy" containerID="cri-o://905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6" gracePeriod=30
Jul 10 18:55:02.489029 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489001    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"]
Jul 10 18:55:02.489304 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489288    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kube-rbac-proxy"
Jul 10 18:55:02.489349 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489307    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kube-rbac-proxy"
Jul 10 18:55:02.489349 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489326    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kserve-container"
Jul 10 18:55:02.489349 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489332    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kserve-container"
Jul 10 18:55:02.489349 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489340    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent"
Jul 10 18:55:02.489349 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489345    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489353    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="storage-initializer"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489359    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="storage-initializer"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489369    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489376    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489384    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489389    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489435    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kube-rbac-proxy"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489445    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kube-rbac-proxy"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489454    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="agent"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489460    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="63753769-9dd1-4e27-9683-e53c43fc12af" containerName="kserve-container"
Jul 10 18:55:02.489532 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.489466    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b128ead-e6ae-471e-be5e-0df7df6150ee" containerName="kserve-container"
Jul 10 18:55:02.491357 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.491342    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.494868 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.494844    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 10 18:55:02.495852 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.495836    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 10 18:55:02.506395 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.506372    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"]
Jul 10 18:55:02.551582 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.551549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.551780 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.551609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbad3675-c51a-422b-b5de-8278e5e3cbd1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.551780 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.551659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.551780 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.551683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdwz\" (UniqueName: \"kubernetes.io/projected/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kube-api-access-bmdwz\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.628802 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.628721    2558 generic.go:356] "Generic (PLEG): container finished" podID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerID="905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6" exitCode=2
Jul 10 18:55:02.628932 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.628795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerDied","Data":"905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6"}
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.652562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdwz\" (UniqueName: \"kubernetes.io/projected/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kube-api-access-bmdwz\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.652682    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.652768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbad3675-c51a-422b-b5de-8278e5e3cbd1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.652816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:55:02.652977    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:55:02.653046    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls podName:dbad3675-c51a-422b-b5de-8278e5e3cbd1 nodeName:}" failed. No retries permitted until 2026-07-10 18:55:03.153024459 +0000 UTC m=+948.414689210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls") pod "isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" (UID: "dbad3675-c51a-422b-b5de-8278e5e3cbd1") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.653792    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.656944 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.654398    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbad3675-c51a-422b-b5de-8278e5e3cbd1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:02.666834 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:02.666810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdwz\" (UniqueName: \"kubernetes.io/projected/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kube-api-access-bmdwz\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:03.157911 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:03.157870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:03.160289 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:03.160259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:03.401495 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:03.401456    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:03.527464 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:03.527433    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"]
Jul 10 18:55:03.530281 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:55:03.530257    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbad3675_c51a_422b_b5de_8278e5e3cbd1.slice/crio-8dfaa793ed73addd9f4d510435f506a11611bc429385586d6fbb0d59291b8fea WatchSource:0}: Error finding container 8dfaa793ed73addd9f4d510435f506a11611bc429385586d6fbb0d59291b8fea: Status 404 returned error can't find the container with id 8dfaa793ed73addd9f4d510435f506a11611bc429385586d6fbb0d59291b8fea
Jul 10 18:55:03.634311 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:03.634276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerStarted","Data":"b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25"}
Jul 10 18:55:03.634495 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:03.634333    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerStarted","Data":"8dfaa793ed73addd9f4d510435f506a11611bc429385586d6fbb0d59291b8fea"}
Jul 10 18:55:05.363290 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:05.363248    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 10 18:55:06.814723 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.814700    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:55:06.888633 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.888552    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bb1fae-d502-4da3-96cb-25de539d8a82-kserve-provision-location\") pod \"48bb1fae-d502-4da3-96cb-25de539d8a82\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") "
Jul 10 18:55:06.888633 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.888616    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bb1fae-d502-4da3-96cb-25de539d8a82-proxy-tls\") pod \"48bb1fae-d502-4da3-96cb-25de539d8a82\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") "
Jul 10 18:55:06.888861 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.888676    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bb1fae-d502-4da3-96cb-25de539d8a82-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"48bb1fae-d502-4da3-96cb-25de539d8a82\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") "
Jul 10 18:55:06.888861 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.888706    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br8ht\" (UniqueName: \"kubernetes.io/projected/48bb1fae-d502-4da3-96cb-25de539d8a82-kube-api-access-br8ht\") pod \"48bb1fae-d502-4da3-96cb-25de539d8a82\" (UID: \"48bb1fae-d502-4da3-96cb-25de539d8a82\") "
Jul 10 18:55:06.888987 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.888960    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bb1fae-d502-4da3-96cb-25de539d8a82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48bb1fae-d502-4da3-96cb-25de539d8a82" (UID: "48bb1fae-d502-4da3-96cb-25de539d8a82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:55:06.889051 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.889029    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bb1fae-d502-4da3-96cb-25de539d8a82-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "48bb1fae-d502-4da3-96cb-25de539d8a82" (UID: "48bb1fae-d502-4da3-96cb-25de539d8a82"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:55:06.890755 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.890724    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48bb1fae-d502-4da3-96cb-25de539d8a82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48bb1fae-d502-4da3-96cb-25de539d8a82" (UID: "48bb1fae-d502-4da3-96cb-25de539d8a82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:55:06.890863 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.890846    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bb1fae-d502-4da3-96cb-25de539d8a82-kube-api-access-br8ht" (OuterVolumeSpecName: "kube-api-access-br8ht") pod "48bb1fae-d502-4da3-96cb-25de539d8a82" (UID: "48bb1fae-d502-4da3-96cb-25de539d8a82"). InnerVolumeSpecName "kube-api-access-br8ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:55:06.989330 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.989287    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bb1fae-d502-4da3-96cb-25de539d8a82-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:55:06.989330 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.989327    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bb1fae-d502-4da3-96cb-25de539d8a82-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:55:06.989330 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.989339    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bb1fae-d502-4da3-96cb-25de539d8a82-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:55:06.989330 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:06.989349    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br8ht\" (UniqueName: \"kubernetes.io/projected/48bb1fae-d502-4da3-96cb-25de539d8a82-kube-api-access-br8ht\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:55:07.648566 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.648533    2558 generic.go:356] "Generic (PLEG): container finished" podID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerID="b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25" exitCode=0
Jul 10 18:55:07.648763 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.648603    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerDied","Data":"b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25"}
Jul 10 18:55:07.650545 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.650520    2558 generic.go:356] "Generic (PLEG): container finished" podID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerID="a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5" exitCode=0
Jul 10 18:55:07.650673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.650557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerDied","Data":"a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5"}
Jul 10 18:55:07.650673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.650577    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"
Jul 10 18:55:07.650673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.650600    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz" event={"ID":"48bb1fae-d502-4da3-96cb-25de539d8a82","Type":"ContainerDied","Data":"d321773a6aa36e21aaec3bb92287d78f73569fed377ddfe35fa16a5213d93ba1"}
Jul 10 18:55:07.650673 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.650627    2558 scope.go:117] "RemoveContainer" containerID="905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6"
Jul 10 18:55:07.659570 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.659550    2558 scope.go:117] "RemoveContainer" containerID="a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5"
Jul 10 18:55:07.671327 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.671306    2558 scope.go:117] "RemoveContainer" containerID="be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5"
Jul 10 18:55:07.684462 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.684435    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"]
Jul 10 18:55:07.685602 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.685446    2558 scope.go:117] "RemoveContainer" containerID="905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6"
Jul 10 18:55:07.685720 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:55:07.685701    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6\": container with ID starting with 905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6 not found: ID does not exist" containerID="905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6"
Jul 10 18:55:07.685811 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.685727    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6"} err="failed to get container status \"905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6\": rpc error: code = NotFound desc = could not find container \"905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6\": container with ID starting with 905e749707c6fc51e0d38f278a253a386f143bb260d7287fb939d11b15692de6 not found: ID does not exist"
Jul 10 18:55:07.685811 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.685768    2558 scope.go:117] "RemoveContainer" containerID="a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5"
Jul 10 18:55:07.686064 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:55:07.686016    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5\": container with ID starting with a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5 not found: ID does not exist" containerID="a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5"
Jul 10 18:55:07.686064 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.686041    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5"} err="failed to get container status \"a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5\": rpc error: code = NotFound desc = could not find container \"a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5\": container with ID starting with a97ba5571d314869abde19801e1eb2208758d00e863acdfa3db6c405327146a5 not found: ID does not exist"
Jul 10 18:55:07.686064 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.686057    2558 scope.go:117] "RemoveContainer" containerID="be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5"
Jul 10 18:55:07.686414 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:55:07.686305    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5\": container with ID starting with be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5 not found: ID does not exist" containerID="be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5"
Jul 10 18:55:07.686488 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.686421    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5"} err="failed to get container status \"be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5\": rpc error: code = NotFound desc = could not find container \"be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5\": container with ID starting with be5ddbf70d45cda092c87c12638e992521b651ce54447326e8279600b9fa01d5 not found: ID does not exist"
Jul 10 18:55:07.688977 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:07.688953    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-57dd68dd5d-nbhqz"]
Jul 10 18:55:08.655728 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:08.655698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerStarted","Data":"1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247"}
Jul 10 18:55:08.656153 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:08.655800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerStarted","Data":"187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30"}
Jul 10 18:55:08.656153 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:08.656093    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:08.656153 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:08.656144    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:08.657142 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:08.657115    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:55:08.681757 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:08.681689    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podStartSLOduration=3.089315368 podStartE2EDuration="6.681676519s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.592361151s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:55:08.679948149 +0000 UTC m=+953.941612969" watchObservedRunningTime="2026-07-10 18:55:08.681676519 +0000 UTC m=+953.943341292"
Jul 10 18:55:09.257328 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:09.257288    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" path="/var/lib/kubelet/pods/48bb1fae-d502-4da3-96cb-25de539d8a82/volumes"
Jul 10 18:55:09.660832 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:09.660727    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:55:14.665968 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:14.665937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:55:14.666652 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:14.666623    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:55:24.666707 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:24.666670    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:55:34.667169 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:34.667130    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:55:44.667417 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:44.667326    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:55:54.666821 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:55:54.666780    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:56:04.667207 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:04.667165    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:56:14.666594 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:14.666556    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:56:24.252922 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:24.252885    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:56:33.177058 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.177019    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"]
Jul 10 18:56:33.177571 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.177273    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" containerID="cri-o://187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30" gracePeriod=30
Jul 10 18:56:33.177571 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.177313    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kube-rbac-proxy" containerID="cri-o://1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247" gracePeriod=30
Jul 10 18:56:33.189082 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189059    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"]
Jul 10 18:56:33.189347 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189330    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container"
Jul 10 18:56:33.189347 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189345    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container"
Jul 10 18:56:33.189461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189354    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kube-rbac-proxy"
Jul 10 18:56:33.189461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189360    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kube-rbac-proxy"
Jul 10 18:56:33.189461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189368    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="storage-initializer"
Jul 10 18:56:33.189461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189373    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="storage-initializer"
Jul 10 18:56:33.189461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189428    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kserve-container"
Jul 10 18:56:33.189461 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.189438    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="48bb1fae-d502-4da3-96cb-25de539d8a82" containerName="kube-rbac-proxy"
Jul 10 18:56:33.192779 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.192763    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.194972 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.194953    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 10 18:56:33.195369 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.195354    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 10 18:56:33.206006 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.205976    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"]
Jul 10 18:56:33.339468 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.339441    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6l5s\" (UniqueName: \"kubernetes.io/projected/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kube-api-access-d6l5s\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.339623 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.339496    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.339623 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.339565    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.339697 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.339622    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a62800-43b9-4b76-8cbf-7b587d8b468e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.440873 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.440786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.440873 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.440834    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.440873 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.440867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a62800-43b9-4b76-8cbf-7b587d8b468e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.441141 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.440904    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6l5s\" (UniqueName: \"kubernetes.io/projected/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kube-api-access-d6l5s\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.441141 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:56:33.441016    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 10 18:56:33.441141 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:56:33.441098    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls podName:c8a62800-43b9-4b76-8cbf-7b587d8b468e nodeName:}" failed. No retries permitted until 2026-07-10 18:56:33.941076817 +0000 UTC m=+1039.202741568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" (UID: "c8a62800-43b9-4b76-8cbf-7b587d8b468e") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 10 18:56:33.441306 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.441225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.441576 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.441558    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a62800-43b9-4b76-8cbf-7b587d8b468e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.448992 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.448967    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6l5s\" (UniqueName: \"kubernetes.io/projected/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kube-api-access-d6l5s\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.945845 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.945812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.948214 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.948185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:33.960404 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.960379    2558 generic.go:356] "Generic (PLEG): container finished" podID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerID="1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247" exitCode=2
Jul 10 18:56:33.960517 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:33.960448    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerDied","Data":"1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247"}
Jul 10 18:56:34.103194 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:34.103148    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:56:34.231641 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:34.231564    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"]
Jul 10 18:56:34.235021 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:56:34.234994    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a62800_43b9_4b76_8cbf_7b587d8b468e.slice/crio-211b832c6c62e10a5958e36621d5d2518453efac44059ad166fa35012365ba45 WatchSource:0}: Error finding container 211b832c6c62e10a5958e36621d5d2518453efac44059ad166fa35012365ba45: Status 404 returned error can't find the container with id 211b832c6c62e10a5958e36621d5d2518453efac44059ad166fa35012365ba45
Jul 10 18:56:34.253303 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:34.253262    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 10 18:56:34.662010 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:34.661961    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 10 18:56:34.965771 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:34.965674    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerStarted","Data":"ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6"}
Jul 10 18:56:34.965771 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:34.965756    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerStarted","Data":"211b832c6c62e10a5958e36621d5d2518453efac44059ad166fa35012365ba45"}
Jul 10 18:56:37.628142 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.628112    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:56:37.772543 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.772513    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmdwz\" (UniqueName: \"kubernetes.io/projected/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kube-api-access-bmdwz\") pod \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") "
Jul 10 18:56:37.772702 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.772575    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls\") pod \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") "
Jul 10 18:56:37.772702 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.772625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbad3675-c51a-422b-b5de-8278e5e3cbd1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") "
Jul 10 18:56:37.772702 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.772667    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kserve-provision-location\") pod \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\" (UID: \"dbad3675-c51a-422b-b5de-8278e5e3cbd1\") "
Jul 10 18:56:37.773108 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.773079    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbad3675-c51a-422b-b5de-8278e5e3cbd1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "dbad3675-c51a-422b-b5de-8278e5e3cbd1" (UID: "dbad3675-c51a-422b-b5de-8278e5e3cbd1"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:56:37.773200 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.773082    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbad3675-c51a-422b-b5de-8278e5e3cbd1" (UID: "dbad3675-c51a-422b-b5de-8278e5e3cbd1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:56:37.774640 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.774621    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kube-api-access-bmdwz" (OuterVolumeSpecName: "kube-api-access-bmdwz") pod "dbad3675-c51a-422b-b5de-8278e5e3cbd1" (UID: "dbad3675-c51a-422b-b5de-8278e5e3cbd1"). InnerVolumeSpecName "kube-api-access-bmdwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:56:37.774704 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.774645    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbad3675-c51a-422b-b5de-8278e5e3cbd1" (UID: "dbad3675-c51a-422b-b5de-8278e5e3cbd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:56:37.873412 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.873386    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmdwz\" (UniqueName: \"kubernetes.io/projected/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kube-api-access-bmdwz\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:56:37.873412 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.873411    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbad3675-c51a-422b-b5de-8278e5e3cbd1-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:56:37.873552 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.873422    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbad3675-c51a-422b-b5de-8278e5e3cbd1-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:56:37.873552 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.873432    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbad3675-c51a-422b-b5de-8278e5e3cbd1-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:56:37.977895 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.977854    2558 generic.go:356] "Generic (PLEG): container finished" podID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerID="187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30" exitCode=0
Jul 10 18:56:37.978019 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.977913    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerDied","Data":"187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30"}
Jul 10 18:56:37.978019 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.977943    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"
Jul 10 18:56:37.978019 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.977965    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr" event={"ID":"dbad3675-c51a-422b-b5de-8278e5e3cbd1","Type":"ContainerDied","Data":"8dfaa793ed73addd9f4d510435f506a11611bc429385586d6fbb0d59291b8fea"}
Jul 10 18:56:37.978019 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.978000    2558 scope.go:117] "RemoveContainer" containerID="1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247"
Jul 10 18:56:37.979529 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.979505    2558 generic.go:356] "Generic (PLEG): container finished" podID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerID="ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6" exitCode=0
Jul 10 18:56:37.979622 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.979563    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerDied","Data":"ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6"}
Jul 10 18:56:37.986955 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.986935    2558 scope.go:117] "RemoveContainer" containerID="187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30"
Jul 10 18:56:37.994213 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:37.994198    2558 scope.go:117] "RemoveContainer" containerID="b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25"
Jul 10 18:56:38.001358 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.001337    2558 scope.go:117] "RemoveContainer" containerID="1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247"
Jul 10 18:56:38.001594 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:56:38.001578    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247\": container with ID starting with 1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247 not found: ID does not exist" containerID="1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247"
Jul 10 18:56:38.001662 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.001600    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247"} err="failed to get container status \"1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247\": rpc error: code = NotFound desc = could not find container \"1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247\": container with ID starting with 1ae9ae8111b3a9b45358378f3e23877536be85927a95c8e05ee1b4e170547247 not found: ID does not exist"
Jul 10 18:56:38.001662 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.001616    2558 scope.go:117] "RemoveContainer" containerID="187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30"
Jul 10 18:56:38.001868 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:56:38.001851    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30\": container with ID starting with 187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30 not found: ID does not exist" containerID="187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30"
Jul 10 18:56:38.001964 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.001871    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30"} err="failed to get container status \"187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30\": rpc error: code = NotFound desc = could not find container \"187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30\": container with ID starting with 187350c5e03f8c7234c0182374d319a941021e05ff9be35c1309e7ab0e5ffd30 not found: ID does not exist"
Jul 10 18:56:38.001964 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.001893    2558 scope.go:117] "RemoveContainer" containerID="b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25"
Jul 10 18:56:38.002138 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:56:38.002120    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25\": container with ID starting with b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25 not found: ID does not exist" containerID="b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25"
Jul 10 18:56:38.002205 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.002142    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25"} err="failed to get container status \"b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25\": rpc error: code = NotFound desc = could not find container \"b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25\": container with ID starting with b4d97d2cc61a727bf30c092ad152800048dbad663eff63a3b94089df0641ff25 not found: ID does not exist"
Jul 10 18:56:38.020710 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.020690    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"]
Jul 10 18:56:38.029329 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:38.029270    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-6d6cc98c84-fd9nr"]
Jul 10 18:56:39.259514 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:56:39.259044    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" path="/var/lib/kubelet/pods/dbad3675-c51a-422b-b5de-8278e5e3cbd1/volumes"
Jul 10 18:58:57.530528 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:58:57.530492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerStarted","Data":"76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47"}
Jul 10 18:58:57.531118 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:58:57.530551    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerStarted","Data":"55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02"}
Jul 10 18:58:57.531118 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:58:57.530789    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:58:57.531118 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:58:57.530849    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:58:57.555757 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:58:57.555674    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" podStartSLOduration=2.161419785 podStartE2EDuration="2m24.555655212s" totalImagesPullingTime="2m18.865581559s" totalInitContainerRuntime="3.528653868s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:58:57.554574947 +0000 UTC m=+1182.816239733" watchObservedRunningTime="2026-07-10 18:58:57.555655212 +0000 UTC m=+1182.817319986"
Jul 10 18:59:03.539777 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:03.539732    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:59:15.237433 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:15.237403    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:59:15.238366 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:15.238343    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 18:59:33.543639 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:33.543604    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:59:43.381157 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.381124    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"]
Jul 10 18:59:43.381571 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.381372    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kserve-container" containerID="cri-o://55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02" gracePeriod=30
Jul 10 18:59:43.381571 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.381425    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kube-rbac-proxy" containerID="cri-o://76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47" gracePeriod=30
Jul 10 18:59:43.403325 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403299    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"]
Jul 10 18:59:43.403593 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403580    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="storage-initializer"
Jul 10 18:59:43.403638 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403595    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="storage-initializer"
Jul 10 18:59:43.403638 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403610    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kube-rbac-proxy"
Jul 10 18:59:43.403638 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403616    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kube-rbac-proxy"
Jul 10 18:59:43.403638 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403623    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container"
Jul 10 18:59:43.403638 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403629    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container"
Jul 10 18:59:43.403851 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403678    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kserve-container"
Jul 10 18:59:43.403851 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.403692    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbad3675-c51a-422b-b5de-8278e5e3cbd1" containerName="kube-rbac-proxy"
Jul 10 18:59:43.409593 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.409563    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.412687 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.412666    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 10 18:59:43.412967 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.412948    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 10 18:59:43.420840 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.420818    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"]
Jul 10 18:59:43.520071 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.520028    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43632919-9ace-47d0-926a-1f8960be3a00-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.520223 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.520087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43632919-9ace-47d0-926a-1f8960be3a00-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.520223 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.520108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlkg\" (UniqueName: \"kubernetes.io/projected/43632919-9ace-47d0-926a-1f8960be3a00-kube-api-access-kmlkg\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.520223 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.520175    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43632919-9ace-47d0-926a-1f8960be3a00-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.535315 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.535267    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 10 18:59:43.621045 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.621005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43632919-9ace-47d0-926a-1f8960be3a00-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.621242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.621064    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43632919-9ace-47d0-926a-1f8960be3a00-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.621242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.621092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlkg\" (UniqueName: \"kubernetes.io/projected/43632919-9ace-47d0-926a-1f8960be3a00-kube-api-access-kmlkg\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.621242 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.621128    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43632919-9ace-47d0-926a-1f8960be3a00-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.621515 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.621493    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43632919-9ace-47d0-926a-1f8960be3a00-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.621909 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.621889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43632919-9ace-47d0-926a-1f8960be3a00-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.623779 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.623725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43632919-9ace-47d0-926a-1f8960be3a00-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.630462 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.630442    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlkg\" (UniqueName: \"kubernetes.io/projected/43632919-9ace-47d0-926a-1f8960be3a00-kube-api-access-kmlkg\") pod \"isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.691925 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.691834    2558 generic.go:356] "Generic (PLEG): container finished" podID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerID="76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47" exitCode=2
Jul 10 18:59:43.692067 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.691911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerDied","Data":"76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47"}
Jul 10 18:59:43.721209 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.721180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:43.851487 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:43.851464    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"]
Jul 10 18:59:43.854103 ip-10-0-140-163 kubenswrapper[2558]: W0710 18:59:43.854079    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43632919_9ace_47d0_926a_1f8960be3a00.slice/crio-87141dd40db4e2da11daafd5c474bb7eb0d875c5f3f666500769aa7e1417ae56 WatchSource:0}: Error finding container 87141dd40db4e2da11daafd5c474bb7eb0d875c5f3f666500769aa7e1417ae56: Status 404 returned error can't find the container with id 87141dd40db4e2da11daafd5c474bb7eb0d875c5f3f666500769aa7e1417ae56
Jul 10 18:59:44.423365 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.423334    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:59:44.527086 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.526996    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls\") pod \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") "
Jul 10 18:59:44.527086 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.527044    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kserve-provision-location\") pod \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") "
Jul 10 18:59:44.527086 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.527070    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6l5s\" (UniqueName: \"kubernetes.io/projected/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kube-api-access-d6l5s\") pod \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") "
Jul 10 18:59:44.527295 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.527120    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a62800-43b9-4b76-8cbf-7b587d8b468e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\" (UID: \"c8a62800-43b9-4b76-8cbf-7b587d8b468e\") "
Jul 10 18:59:44.527462 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.527418    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8a62800-43b9-4b76-8cbf-7b587d8b468e" (UID: "c8a62800-43b9-4b76-8cbf-7b587d8b468e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 18:59:44.527462 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.527445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a62800-43b9-4b76-8cbf-7b587d8b468e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "c8a62800-43b9-4b76-8cbf-7b587d8b468e" (UID: "c8a62800-43b9-4b76-8cbf-7b587d8b468e"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 18:59:44.529121 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.529098    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kube-api-access-d6l5s" (OuterVolumeSpecName: "kube-api-access-d6l5s") pod "c8a62800-43b9-4b76-8cbf-7b587d8b468e" (UID: "c8a62800-43b9-4b76-8cbf-7b587d8b468e"). InnerVolumeSpecName "kube-api-access-d6l5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 18:59:44.529218 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.529201    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8a62800-43b9-4b76-8cbf-7b587d8b468e" (UID: "c8a62800-43b9-4b76-8cbf-7b587d8b468e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 18:59:44.628038 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.628004    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8a62800-43b9-4b76-8cbf-7b587d8b468e-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:59:44.628038 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.628034    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:59:44.628038 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.628045    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6l5s\" (UniqueName: \"kubernetes.io/projected/c8a62800-43b9-4b76-8cbf-7b587d8b468e-kube-api-access-d6l5s\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:59:44.628266 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.628055    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8a62800-43b9-4b76-8cbf-7b587d8b468e-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 18:59:44.696725 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.696692    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerStarted","Data":"0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2"}
Jul 10 18:59:44.696906 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.696780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerStarted","Data":"87141dd40db4e2da11daafd5c474bb7eb0d875c5f3f666500769aa7e1417ae56"}
Jul 10 18:59:44.698509 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.698486    2558 generic.go:356] "Generic (PLEG): container finished" podID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerID="55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02" exitCode=0
Jul 10 18:59:44.698609 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.698532    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"
Jul 10 18:59:44.698666 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.698532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerDied","Data":"55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02"}
Jul 10 18:59:44.698726 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.698679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm" event={"ID":"c8a62800-43b9-4b76-8cbf-7b587d8b468e","Type":"ContainerDied","Data":"211b832c6c62e10a5958e36621d5d2518453efac44059ad166fa35012365ba45"}
Jul 10 18:59:44.698799 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.698728    2558 scope.go:117] "RemoveContainer" containerID="76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47"
Jul 10 18:59:44.707208 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.707193    2558 scope.go:117] "RemoveContainer" containerID="55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02"
Jul 10 18:59:44.713959 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.713942    2558 scope.go:117] "RemoveContainer" containerID="ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6"
Jul 10 18:59:44.721523 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.721505    2558 scope.go:117] "RemoveContainer" containerID="76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47"
Jul 10 18:59:44.721796 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:59:44.721777    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47\": container with ID starting with 76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47 not found: ID does not exist" containerID="76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47"
Jul 10 18:59:44.721858 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.721804    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47"} err="failed to get container status \"76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47\": rpc error: code = NotFound desc = could not find container \"76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47\": container with ID starting with 76b5adaea2a693966ae549832a741529461a8062a7818cba99b300e0052f3b47 not found: ID does not exist"
Jul 10 18:59:44.721858 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.721823    2558 scope.go:117] "RemoveContainer" containerID="55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02"
Jul 10 18:59:44.722070 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:59:44.722052    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02\": container with ID starting with 55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02 not found: ID does not exist" containerID="55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02"
Jul 10 18:59:44.722117 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.722075    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02"} err="failed to get container status \"55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02\": rpc error: code = NotFound desc = could not find container \"55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02\": container with ID starting with 55b90565ca5776f600378ed8c7cabe41d92109cb4ebda72fce462d39b21d0d02 not found: ID does not exist"
Jul 10 18:59:44.722117 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.722091    2558 scope.go:117] "RemoveContainer" containerID="ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6"
Jul 10 18:59:44.722339 ip-10-0-140-163 kubenswrapper[2558]: E0710 18:59:44.722318    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6\": container with ID starting with ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6 not found: ID does not exist" containerID="ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6"
Jul 10 18:59:44.722388 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.722344    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6"} err="failed to get container status \"ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6\": rpc error: code = NotFound desc = could not find container \"ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6\": container with ID starting with ede3f5c158706a7656ef42af0b30bf56edb7c050afd424b02c3385fc6efc70e6 not found: ID does not exist"
Jul 10 18:59:44.728802 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.728781    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"]
Jul 10 18:59:44.734804 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:44.734779    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-5c56b6d86c-7v7vm"]
Jul 10 18:59:45.257584 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:45.257552    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" path="/var/lib/kubelet/pods/c8a62800-43b9-4b76-8cbf-7b587d8b468e/volumes"
Jul 10 18:59:47.714581 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:47.714490    2558 generic.go:356] "Generic (PLEG): container finished" podID="43632919-9ace-47d0-926a-1f8960be3a00" containerID="0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2" exitCode=0
Jul 10 18:59:47.715016 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:47.714575    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerDied","Data":"0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2"}
Jul 10 18:59:48.721019 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:48.720988    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerStarted","Data":"e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411"}
Jul 10 18:59:48.721481 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:48.721057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerStarted","Data":"206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4"}
Jul 10 18:59:48.721481 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:48.721368    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:48.721481 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:48.721417    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:48.722953 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:48.722920    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:59:48.742603 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:48.742553    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" podStartSLOduration=2.252954567 podStartE2EDuration="5.742534131s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.489579564s" isStatefulPod=false podCreationTimestamp="2026-07-10 18:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 18:59:48.742025634 +0000 UTC m=+1234.003690407" watchObservedRunningTime="2026-07-10 18:59:48.742534131 +0000 UTC m=+1234.004198904"
Jul 10 18:59:49.724637 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:49.724594    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 18:59:54.729314 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:54.729286    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 18:59:54.729911 ip-10-0-140-163 kubenswrapper[2558]: I0710 18:59:54.729879    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 10 19:00:04.731210 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:04.731178    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 19:00:13.534946 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.534830    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"]
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535214    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="storage-initializer"
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535234    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="storage-initializer"
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535253    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kube-rbac-proxy"
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535262    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kube-rbac-proxy"
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535281    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kserve-container"
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535289    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kserve-container"
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535345    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kserve-container"
Jul 10 19:00:13.535390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.535358    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a62800-43b9-4b76-8cbf-7b587d8b468e" containerName="kube-rbac-proxy"
Jul 10 19:00:13.537828 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.537808    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.540254 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.540232    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 10 19:00:13.540366 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.540240    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:00:13.554690 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.554662    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"]
Jul 10 19:00:13.587563 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.587530    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"]
Jul 10 19:00:13.587802 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.587774    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kserve-container" containerID="cri-o://206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4" gracePeriod=30
Jul 10 19:00:13.587931 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.587794    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kube-rbac-proxy" containerID="cri-o://e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411" gracePeriod=30
Jul 10 19:00:13.637030 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.636998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0a2ae6f-8983-4a76-a982-703e12a35ddc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.637151 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.637035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.637151 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.637058    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbth\" (UniqueName: \"kubernetes.io/projected/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kube-api-access-4rbth\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.637151 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.637132    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0a2ae6f-8983-4a76-a982-703e12a35ddc-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.738472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.738427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0a2ae6f-8983-4a76-a982-703e12a35ddc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.738668 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.738487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.738668 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.738525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbth\" (UniqueName: \"kubernetes.io/projected/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kube-api-access-4rbth\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.738668 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.738551    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0a2ae6f-8983-4a76-a982-703e12a35ddc-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.739033 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.739009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.739398 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.739377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0a2ae6f-8983-4a76-a982-703e12a35ddc-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.741034 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.741016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0a2ae6f-8983-4a76-a982-703e12a35ddc-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.753552 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.753532    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbth\" (UniqueName: \"kubernetes.io/projected/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kube-api-access-4rbth\") pod \"isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.822096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.822020    2558 generic.go:356] "Generic (PLEG): container finished" podID="43632919-9ace-47d0-926a-1f8960be3a00" containerID="e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411" exitCode=2
Jul 10 19:00:13.822233 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.822095    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerDied","Data":"e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411"}
Jul 10 19:00:13.847367 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.847337    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:13.981203 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:13.981182    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"]
Jul 10 19:00:13.983562 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:00:13.983532    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a2ae6f_8983_4a76_a982_703e12a35ddc.slice/crio-f6b61911b4cafca7711ffcca5da89c868c7bd94e989a440c299eff589d4975f2 WatchSource:0}: Error finding container f6b61911b4cafca7711ffcca5da89c868c7bd94e989a440c299eff589d4975f2: Status 404 returned error can't find the container with id f6b61911b4cafca7711ffcca5da89c868c7bd94e989a440c299eff589d4975f2
Jul 10 19:00:14.219262 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.219234    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 19:00:14.343491 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.343463    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43632919-9ace-47d0-926a-1f8960be3a00-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") pod \"43632919-9ace-47d0-926a-1f8960be3a00\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") "
Jul 10 19:00:14.343640 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.343502    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmlkg\" (UniqueName: \"kubernetes.io/projected/43632919-9ace-47d0-926a-1f8960be3a00-kube-api-access-kmlkg\") pod \"43632919-9ace-47d0-926a-1f8960be3a00\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") "
Jul 10 19:00:14.343640 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.343558    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43632919-9ace-47d0-926a-1f8960be3a00-proxy-tls\") pod \"43632919-9ace-47d0-926a-1f8960be3a00\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") "
Jul 10 19:00:14.343640 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.343596    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43632919-9ace-47d0-926a-1f8960be3a00-kserve-provision-location\") pod \"43632919-9ace-47d0-926a-1f8960be3a00\" (UID: \"43632919-9ace-47d0-926a-1f8960be3a00\") "
Jul 10 19:00:14.343963 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.343934    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43632919-9ace-47d0-926a-1f8960be3a00-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config") pod "43632919-9ace-47d0-926a-1f8960be3a00" (UID: "43632919-9ace-47d0-926a-1f8960be3a00"). InnerVolumeSpecName "isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:00:14.344089 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.344060    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43632919-9ace-47d0-926a-1f8960be3a00-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43632919-9ace-47d0-926a-1f8960be3a00" (UID: "43632919-9ace-47d0-926a-1f8960be3a00"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:00:14.345613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.345584    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43632919-9ace-47d0-926a-1f8960be3a00-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43632919-9ace-47d0-926a-1f8960be3a00" (UID: "43632919-9ace-47d0-926a-1f8960be3a00"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:00:14.345702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.345618    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43632919-9ace-47d0-926a-1f8960be3a00-kube-api-access-kmlkg" (OuterVolumeSpecName: "kube-api-access-kmlkg") pod "43632919-9ace-47d0-926a-1f8960be3a00" (UID: "43632919-9ace-47d0-926a-1f8960be3a00"). InnerVolumeSpecName "kube-api-access-kmlkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:00:14.444201 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.444129    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43632919-9ace-47d0-926a-1f8960be3a00-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:00:14.444201 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.444154    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43632919-9ace-47d0-926a-1f8960be3a00-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:00:14.444201 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.444166    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43632919-9ace-47d0-926a-1f8960be3a00-isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:00:14.444201 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.444176    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmlkg\" (UniqueName: \"kubernetes.io/projected/43632919-9ace-47d0-926a-1f8960be3a00-kube-api-access-kmlkg\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:00:14.827917 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.827878    2558 generic.go:356] "Generic (PLEG): container finished" podID="43632919-9ace-47d0-926a-1f8960be3a00" containerID="206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4" exitCode=0
Jul 10 19:00:14.828346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.827956    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"
Jul 10 19:00:14.828346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.827971    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerDied","Data":"206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4"}
Jul 10 19:00:14.828346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.828061    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls" event={"ID":"43632919-9ace-47d0-926a-1f8960be3a00","Type":"ContainerDied","Data":"87141dd40db4e2da11daafd5c474bb7eb0d875c5f3f666500769aa7e1417ae56"}
Jul 10 19:00:14.828346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.828093    2558 scope.go:117] "RemoveContainer" containerID="e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411"
Jul 10 19:00:14.829600 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.829578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerStarted","Data":"409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9"}
Jul 10 19:00:14.829697 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.829671    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerStarted","Data":"f6b61911b4cafca7711ffcca5da89c868c7bd94e989a440c299eff589d4975f2"}
Jul 10 19:00:14.836727 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.836709    2558 scope.go:117] "RemoveContainer" containerID="206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4"
Jul 10 19:00:14.843569 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.843550    2558 scope.go:117] "RemoveContainer" containerID="0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2"
Jul 10 19:00:14.850074 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.850056    2558 scope.go:117] "RemoveContainer" containerID="e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411"
Jul 10 19:00:14.850311 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:00:14.850292    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411\": container with ID starting with e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411 not found: ID does not exist" containerID="e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411"
Jul 10 19:00:14.850361 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.850320    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411"} err="failed to get container status \"e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411\": rpc error: code = NotFound desc = could not find container \"e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411\": container with ID starting with e39dc7f2768dc9c81c9701d5b6ae7cbc93db1611d376a80465d2f2fead2b9411 not found: ID does not exist"
Jul 10 19:00:14.850361 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.850340    2558 scope.go:117] "RemoveContainer" containerID="206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4"
Jul 10 19:00:14.850576 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:00:14.850560    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4\": container with ID starting with 206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4 not found: ID does not exist" containerID="206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4"
Jul 10 19:00:14.850618 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.850582    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4"} err="failed to get container status \"206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4\": rpc error: code = NotFound desc = could not find container \"206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4\": container with ID starting with 206ae04503d3592eb5983612e4e08f24045d9cd02d8bbce29a5012a25ba3a6b4 not found: ID does not exist"
Jul 10 19:00:14.850618 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.850600    2558 scope.go:117] "RemoveContainer" containerID="0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2"
Jul 10 19:00:14.850831 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:00:14.850812    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2\": container with ID starting with 0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2 not found: ID does not exist" containerID="0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2"
Jul 10 19:00:14.850888 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.850839    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2"} err="failed to get container status \"0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2\": rpc error: code = NotFound desc = could not find container \"0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2\": container with ID starting with 0a9ec6aa1ae6c427255c18ad91fe28753079f807cc6cabd1d757d6f3ed9202c2 not found: ID does not exist"
Jul 10 19:00:14.866621 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.866598    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"]
Jul 10 19:00:14.870957 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:14.870932    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-745bd46555-mncls"]
Jul 10 19:00:15.258174 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:15.258144    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43632919-9ace-47d0-926a-1f8960be3a00" path="/var/lib/kubelet/pods/43632919-9ace-47d0-926a-1f8960be3a00/volumes"
Jul 10 19:00:18.847038 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:18.847003    2558 generic.go:356] "Generic (PLEG): container finished" podID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerID="409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9" exitCode=0
Jul 10 19:00:18.847477 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:18.847056    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerDied","Data":"409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9"}
Jul 10 19:00:19.853391 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:19.853359    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerStarted","Data":"de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad"}
Jul 10 19:00:19.853793 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:19.853420    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerStarted","Data":"9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8"}
Jul 10 19:00:19.853793 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:19.853623    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:19.875690 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:19.875626    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" podStartSLOduration=3.043037414 podStartE2EDuration="6.875606197s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.832568783s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:19.874391091 +0000 UTC m=+1265.136055892" watchObservedRunningTime="2026-07-10 19:00:19.875606197 +0000 UTC m=+1265.137270971"
Jul 10 19:00:20.857913 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:20.857878    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:26.867120 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:26.867092    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:00:56.871711 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:00:56.871680    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:01:03.649697 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.649644    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"]
Jul 10 19:01:03.650127 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.649957    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kserve-container" containerID="cri-o://9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8" gracePeriod=30
Jul 10 19:01:03.650127 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.650003    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kube-rbac-proxy" containerID="cri-o://de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad" gracePeriod=30
Jul 10 19:01:03.732657 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.732599    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"]
Jul 10 19:01:03.733065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733046    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kube-rbac-proxy"
Jul 10 19:01:03.733111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733069    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kube-rbac-proxy"
Jul 10 19:01:03.733111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733095    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kserve-container"
Jul 10 19:01:03.733111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733104    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kserve-container"
Jul 10 19:01:03.733209 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733117    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="storage-initializer"
Jul 10 19:01:03.733209 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733129    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="storage-initializer"
Jul 10 19:01:03.733209 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733200    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kserve-container"
Jul 10 19:01:03.733300 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.733210    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="43632919-9ace-47d0-926a-1f8960be3a00" containerName="kube-rbac-proxy"
Jul 10 19:01:03.735916 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.735895    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.739191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.739166    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 10 19:01:03.739472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.739447    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 10 19:01:03.749414 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.749387    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"]
Jul 10 19:01:03.810393 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.810359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f203187-df69-4531-ae76-ac5a3449a8be-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.810580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.810403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f203187-df69-4531-ae76-ac5a3449a8be-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.810580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.810426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f203187-df69-4531-ae76-ac5a3449a8be-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.810580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.810498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8jpb\" (UniqueName: \"kubernetes.io/projected/7f203187-df69-4531-ae76-ac5a3449a8be-kube-api-access-r8jpb\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.911328 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.911220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f203187-df69-4531-ae76-ac5a3449a8be-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.911328 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.911277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f203187-df69-4531-ae76-ac5a3449a8be-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.911557 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.911404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f203187-df69-4531-ae76-ac5a3449a8be-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.911557 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.911466    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8jpb\" (UniqueName: \"kubernetes.io/projected/7f203187-df69-4531-ae76-ac5a3449a8be-kube-api-access-r8jpb\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.911715 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.911692    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f203187-df69-4531-ae76-ac5a3449a8be-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.912110 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.912092    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f203187-df69-4531-ae76-ac5a3449a8be-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.914100 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.914076    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f203187-df69-4531-ae76-ac5a3449a8be-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:03.921382 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:03.921351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8jpb\" (UniqueName: \"kubernetes.io/projected/7f203187-df69-4531-ae76-ac5a3449a8be-kube-api-access-r8jpb\") pod \"isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:04.005172 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.005140    2558 generic.go:356] "Generic (PLEG): container finished" podID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerID="de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad" exitCode=2
Jul 10 19:01:04.005341 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.005218    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerDied","Data":"de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad"}
Jul 10 19:01:04.046765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.046714    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:04.189579 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.189556    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"]
Jul 10 19:01:04.191447 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:01:04.191419    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f203187_df69_4531_ae76_ac5a3449a8be.slice/crio-6f3514a481515e1d5c2d261a51c50aabd832b59e4ffa220037a78eeba556b0ff WatchSource:0}: Error finding container 6f3514a481515e1d5c2d261a51c50aabd832b59e4ffa220037a78eeba556b0ff: Status 404 returned error can't find the container with id 6f3514a481515e1d5c2d261a51c50aabd832b59e4ffa220037a78eeba556b0ff
Jul 10 19:01:04.891995 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.891974    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:01:04.919824 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.919798    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0a2ae6f-8983-4a76-a982-703e12a35ddc-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") "
Jul 10 19:01:04.919937 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.919855    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rbth\" (UniqueName: \"kubernetes.io/projected/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kube-api-access-4rbth\") pod \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") "
Jul 10 19:01:04.919937 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.919887    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0a2ae6f-8983-4a76-a982-703e12a35ddc-proxy-tls\") pod \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") "
Jul 10 19:01:04.919937 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.919917    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kserve-provision-location\") pod \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\" (UID: \"c0a2ae6f-8983-4a76-a982-703e12a35ddc\") "
Jul 10 19:01:04.920223 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.920189    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a2ae6f-8983-4a76-a982-703e12a35ddc-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "c0a2ae6f-8983-4a76-a982-703e12a35ddc" (UID: "c0a2ae6f-8983-4a76-a982-703e12a35ddc"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:01:04.920335 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.920264    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0a2ae6f-8983-4a76-a982-703e12a35ddc" (UID: "c0a2ae6f-8983-4a76-a982-703e12a35ddc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:01:04.922064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.922037    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a2ae6f-8983-4a76-a982-703e12a35ddc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0a2ae6f-8983-4a76-a982-703e12a35ddc" (UID: "c0a2ae6f-8983-4a76-a982-703e12a35ddc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:01:04.922313 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:04.922285    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kube-api-access-4rbth" (OuterVolumeSpecName: "kube-api-access-4rbth") pod "c0a2ae6f-8983-4a76-a982-703e12a35ddc" (UID: "c0a2ae6f-8983-4a76-a982-703e12a35ddc"). InnerVolumeSpecName "kube-api-access-4rbth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:01:05.010353 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.010321    2558 generic.go:356] "Generic (PLEG): container finished" podID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerID="9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8" exitCode=0
Jul 10 19:01:05.010518 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.010380    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"
Jul 10 19:01:05.010518 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.010396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerDied","Data":"9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8"}
Jul 10 19:01:05.010518 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.010453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb" event={"ID":"c0a2ae6f-8983-4a76-a982-703e12a35ddc","Type":"ContainerDied","Data":"f6b61911b4cafca7711ffcca5da89c868c7bd94e989a440c299eff589d4975f2"}
Jul 10 19:01:05.010518 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.010486    2558 scope.go:117] "RemoveContainer" containerID="de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad"
Jul 10 19:01:05.012038 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.012018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerStarted","Data":"6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39"}
Jul 10 19:01:05.012162 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.012063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerStarted","Data":"6f3514a481515e1d5c2d261a51c50aabd832b59e4ffa220037a78eeba556b0ff"}
Jul 10 19:01:05.018881 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.018866    2558 scope.go:117] "RemoveContainer" containerID="9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8"
Jul 10 19:01:05.020610 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.020590    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:01:05.020703 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.020614    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0a2ae6f-8983-4a76-a982-703e12a35ddc-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:01:05.020703 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.020630    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rbth\" (UniqueName: \"kubernetes.io/projected/c0a2ae6f-8983-4a76-a982-703e12a35ddc-kube-api-access-4rbth\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:01:05.020703 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.020643    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0a2ae6f-8983-4a76-a982-703e12a35ddc-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:01:05.025981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.025965    2558 scope.go:117] "RemoveContainer" containerID="409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9"
Jul 10 19:01:05.032352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.032332    2558 scope.go:117] "RemoveContainer" containerID="de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad"
Jul 10 19:01:05.032595 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:01:05.032571    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad\": container with ID starting with de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad not found: ID does not exist" containerID="de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad"
Jul 10 19:01:05.032645 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.032598    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad"} err="failed to get container status \"de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad\": rpc error: code = NotFound desc = could not find container \"de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad\": container with ID starting with de376edc874e4a9f38fd0309feb22bd64e524a79b5a6d317ce23f173874272ad not found: ID does not exist"
Jul 10 19:01:05.032645 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.032616    2558 scope.go:117] "RemoveContainer" containerID="9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8"
Jul 10 19:01:05.032881 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:01:05.032862    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8\": container with ID starting with 9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8 not found: ID does not exist" containerID="9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8"
Jul 10 19:01:05.032927 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.032887    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8"} err="failed to get container status \"9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8\": rpc error: code = NotFound desc = could not find container \"9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8\": container with ID starting with 9d8a192a3adc59f5ec8edda598bf46f5366de5d1ef57e6a2a94d32db0d9433e8 not found: ID does not exist"
Jul 10 19:01:05.032927 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.032902    2558 scope.go:117] "RemoveContainer" containerID="409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9"
Jul 10 19:01:05.033127 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:01:05.033112    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9\": container with ID starting with 409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9 not found: ID does not exist" containerID="409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9"
Jul 10 19:01:05.033169 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.033131    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9"} err="failed to get container status \"409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9\": rpc error: code = NotFound desc = could not find container \"409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9\": container with ID starting with 409065efc4f76a7d10c090609070a6f36e8de9bc7fb426a0fed3d23d4ece62b9 not found: ID does not exist"
Jul 10 19:01:05.054241 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.054216    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"]
Jul 10 19:01:05.060584 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.060561    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-b94c64cf4-4k2sb"]
Jul 10 19:01:05.260690 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:05.260611    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" path="/var/lib/kubelet/pods/c0a2ae6f-8983-4a76-a982-703e12a35ddc/volumes"
Jul 10 19:01:08.024655 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:08.024574    2558 generic.go:356] "Generic (PLEG): container finished" podID="7f203187-df69-4531-ae76-ac5a3449a8be" containerID="6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39" exitCode=0
Jul 10 19:01:08.025016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:08.024646    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerDied","Data":"6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39"}
Jul 10 19:01:08.079423 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:08.079403    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:01:09.030711 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:09.030673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerStarted","Data":"f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7"}
Jul 10 19:01:11.045402 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:11.045369    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerStarted","Data":"aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e"}
Jul 10 19:01:11.045812 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:11.045465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerStarted","Data":"0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782"}
Jul 10 19:01:11.045812 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:11.045502    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:11.045812 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:11.045528    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:11.070851 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:11.070800    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podStartSLOduration=1.760877987 podStartE2EDuration="8.07078413s" totalImagesPullingTime="2.789111756s" totalInitContainerRuntime="3.520794387s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:11.069510833 +0000 UTC m=+1316.331175606" watchObservedRunningTime="2026-07-10 19:01:11.07078413 +0000 UTC m=+1316.332448903"
Jul 10 19:01:12.054210 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:12.054176    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:18.062666 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:18.062635    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:01:38.063622 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:38.063535    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 19:01:48.065379 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:01:48.065346    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:02:18.066408 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:18.066378    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:02:23.830824 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.830792    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"]
Jul 10 19:02:23.831308 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.831065    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container" containerID="cri-o://f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7" gracePeriod=30
Jul 10 19:02:23.831308 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.831119    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-agent" containerID="cri-o://0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782" gracePeriod=30
Jul 10 19:02:23.831308 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.831127    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy" containerID="cri-o://aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e" gracePeriod=30
Jul 10 19:02:23.918933 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.918900    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"]
Jul 10 19:02:23.919178 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919165    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="storage-initializer"
Jul 10 19:02:23.919221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919179    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="storage-initializer"
Jul 10 19:02:23.919221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919188    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kserve-container"
Jul 10 19:02:23.919221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919194    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kserve-container"
Jul 10 19:02:23.919221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919201    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kube-rbac-proxy"
Jul 10 19:02:23.919221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919207    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kube-rbac-proxy"
Jul 10 19:02:23.919368 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919245    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kserve-container"
Jul 10 19:02:23.919368 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.919252    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a2ae6f-8983-4a76-a982-703e12a35ddc" containerName="kube-rbac-proxy"
Jul 10 19:02:23.921301 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.921272    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:23.924402 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.924377    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 10 19:02:23.924951 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.924937    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 10 19:02:23.938038 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.938013    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"]
Jul 10 19:02:23.994134 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.994111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd7e8d76-1089-427b-b386-8b0ce820e596-proxy-tls\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:23.994289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.994179    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbzml\" (UniqueName: \"kubernetes.io/projected/bd7e8d76-1089-427b-b386-8b0ce820e596-kube-api-access-cbzml\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:23.994289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.994213    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7e8d76-1089-427b-b386-8b0ce820e596-kserve-provision-location\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:23.994289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:23.994235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd7e8d76-1089-427b-b386-8b0ce820e596-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.094645 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.094555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbzml\" (UniqueName: \"kubernetes.io/projected/bd7e8d76-1089-427b-b386-8b0ce820e596-kube-api-access-cbzml\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.094645 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.094604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7e8d76-1089-427b-b386-8b0ce820e596-kserve-provision-location\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.094896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.094709    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd7e8d76-1089-427b-b386-8b0ce820e596-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.094896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.094793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd7e8d76-1089-427b-b386-8b0ce820e596-proxy-tls\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.095151 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.095129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7e8d76-1089-427b-b386-8b0ce820e596-kserve-provision-location\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.095500 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.095483    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd7e8d76-1089-427b-b386-8b0ce820e596-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.097184 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.097163    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd7e8d76-1089-427b-b386-8b0ce820e596-proxy-tls\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.105473 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.105438    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbzml\" (UniqueName: \"kubernetes.io/projected/bd7e8d76-1089-427b-b386-8b0ce820e596-kube-api-access-cbzml\") pod \"isvc-paddle-predictor-574f94bdcb-h24rm\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.231513 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.231477    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:24.317001 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.316965    2558 generic.go:356] "Generic (PLEG): container finished" podID="7f203187-df69-4531-ae76-ac5a3449a8be" containerID="aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e" exitCode=2
Jul 10 19:02:24.317148 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.317013    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerDied","Data":"aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e"}
Jul 10 19:02:24.363878 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:24.363854    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"]
Jul 10 19:02:24.364458 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:02:24.364427    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7e8d76_1089_427b_b386_8b0ce820e596.slice/crio-a0ca70d898e2f2dbc984719b629e2895cdbe981f7e67b10273d6cb4bbaadaea0 WatchSource:0}: Error finding container a0ca70d898e2f2dbc984719b629e2895cdbe981f7e67b10273d6cb4bbaadaea0: Status 404 returned error can't find the container with id a0ca70d898e2f2dbc984719b629e2895cdbe981f7e67b10273d6cb4bbaadaea0
Jul 10 19:02:25.321786 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:25.321728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerStarted","Data":"b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea"}
Jul 10 19:02:25.322187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:25.321806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerStarted","Data":"a0ca70d898e2f2dbc984719b629e2895cdbe981f7e67b10273d6cb4bbaadaea0"}
Jul 10 19:02:26.326973 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:26.326941    2558 generic.go:356] "Generic (PLEG): container finished" podID="7f203187-df69-4531-ae76-ac5a3449a8be" containerID="f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7" exitCode=0
Jul 10 19:02:26.327346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:26.327008    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerDied","Data":"f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7"}
Jul 10 19:02:28.058893 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:28.058848    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 19:02:28.064129 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:28.064107    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 19:02:29.339887 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:29.339855    2558 generic.go:356] "Generic (PLEG): container finished" podID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerID="b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea" exitCode=0
Jul 10 19:02:29.340229 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:29.339914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerDied","Data":"b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea"}
Jul 10 19:02:33.058667 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:33.058620    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 19:02:38.058851 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:38.058800    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 19:02:38.059318 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:38.059009    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:02:38.063348 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:38.063311    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 19:02:42.395469 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:42.395437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerStarted","Data":"a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb"}
Jul 10 19:02:42.395905 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:42.395498    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerStarted","Data":"28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18"}
Jul 10 19:02:42.395905 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:42.395703    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:42.422163 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:42.422116    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podStartSLOduration=2.539508442 podStartE2EDuration="19.422100157s" totalImagesPullingTime="11.992341259s" totalInitContainerRuntime="4.890250456s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:02:42.419709558 +0000 UTC m=+1407.681374333" watchObservedRunningTime="2026-07-10 19:02:42.422100157 +0000 UTC m=+1407.683764930"
Jul 10 19:02:43.058593 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:43.058551    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 19:02:43.399804 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:43.399724    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:43.400790 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:43.400762    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 19:02:44.403229 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:44.403191    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 19:02:48.058813 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:48.058762    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 19:02:48.064099 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:48.064072    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 10 19:02:48.064215 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:48.064204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:02:49.409412 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:49.409382    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:02:49.409990 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:49.409955    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 19:02:53.058859 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:53.058820    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 10 19:02:54.023728 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.023696    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:02:54.139249 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.139164    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f203187-df69-4531-ae76-ac5a3449a8be-kserve-provision-location\") pod \"7f203187-df69-4531-ae76-ac5a3449a8be\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") "
Jul 10 19:02:54.139249 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.139215    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f203187-df69-4531-ae76-ac5a3449a8be-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"7f203187-df69-4531-ae76-ac5a3449a8be\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") "
Jul 10 19:02:54.139249 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.139244    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f203187-df69-4531-ae76-ac5a3449a8be-proxy-tls\") pod \"7f203187-df69-4531-ae76-ac5a3449a8be\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") "
Jul 10 19:02:54.139812 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.139268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8jpb\" (UniqueName: \"kubernetes.io/projected/7f203187-df69-4531-ae76-ac5a3449a8be-kube-api-access-r8jpb\") pod \"7f203187-df69-4531-ae76-ac5a3449a8be\" (UID: \"7f203187-df69-4531-ae76-ac5a3449a8be\") "
Jul 10 19:02:54.139812 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.139470    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f203187-df69-4531-ae76-ac5a3449a8be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f203187-df69-4531-ae76-ac5a3449a8be" (UID: "7f203187-df69-4531-ae76-ac5a3449a8be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:02:54.139812 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.139665    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f203187-df69-4531-ae76-ac5a3449a8be-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "7f203187-df69-4531-ae76-ac5a3449a8be" (UID: "7f203187-df69-4531-ae76-ac5a3449a8be"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:02:54.141390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.141370    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f203187-df69-4531-ae76-ac5a3449a8be-kube-api-access-r8jpb" (OuterVolumeSpecName: "kube-api-access-r8jpb") pod "7f203187-df69-4531-ae76-ac5a3449a8be" (UID: "7f203187-df69-4531-ae76-ac5a3449a8be"). InnerVolumeSpecName "kube-api-access-r8jpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:02:54.141491 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.141467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f203187-df69-4531-ae76-ac5a3449a8be-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f203187-df69-4531-ae76-ac5a3449a8be" (UID: "7f203187-df69-4531-ae76-ac5a3449a8be"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:02:54.240071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.240030    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8jpb\" (UniqueName: \"kubernetes.io/projected/7f203187-df69-4531-ae76-ac5a3449a8be-kube-api-access-r8jpb\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:02:54.240071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.240064    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f203187-df69-4531-ae76-ac5a3449a8be-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:02:54.240071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.240075    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f203187-df69-4531-ae76-ac5a3449a8be-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:02:54.240285 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.240086    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f203187-df69-4531-ae76-ac5a3449a8be-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:02:54.446123 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.446041    2558 generic.go:356] "Generic (PLEG): container finished" podID="7f203187-df69-4531-ae76-ac5a3449a8be" containerID="0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782" exitCode=0
Jul 10 19:02:54.446289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.446122    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerDied","Data":"0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782"}
Jul 10 19:02:54.446289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.446146    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"
Jul 10 19:02:54.446289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.446191    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf" event={"ID":"7f203187-df69-4531-ae76-ac5a3449a8be","Type":"ContainerDied","Data":"6f3514a481515e1d5c2d261a51c50aabd832b59e4ffa220037a78eeba556b0ff"}
Jul 10 19:02:54.446289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.446222    2558 scope.go:117] "RemoveContainer" containerID="aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e"
Jul 10 19:02:54.454424 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.454329    2558 scope.go:117] "RemoveContainer" containerID="0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782"
Jul 10 19:02:54.461522 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.461506    2558 scope.go:117] "RemoveContainer" containerID="f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7"
Jul 10 19:02:54.468182 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.468165    2558 scope.go:117] "RemoveContainer" containerID="6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39"
Jul 10 19:02:54.474883 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.474862    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"]
Jul 10 19:02:54.475084 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.475070    2558 scope.go:117] "RemoveContainer" containerID="aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e"
Jul 10 19:02:54.475555 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:02:54.475514    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e\": container with ID starting with aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e not found: ID does not exist" containerID="aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e"
Jul 10 19:02:54.475653 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.475569    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e"} err="failed to get container status \"aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e\": rpc error: code = NotFound desc = could not find container \"aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e\": container with ID starting with aa4c356e7803fe87e1e9e77ab352542c47589125af696e0f1b72bb71cbe4327e not found: ID does not exist"
Jul 10 19:02:54.475653 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.475601    2558 scope.go:117] "RemoveContainer" containerID="0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782"
Jul 10 19:02:54.476020 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:02:54.475895    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782\": container with ID starting with 0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782 not found: ID does not exist" containerID="0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782"
Jul 10 19:02:54.476020 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.475928    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782"} err="failed to get container status \"0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782\": rpc error: code = NotFound desc = could not find container \"0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782\": container with ID starting with 0a40215c00b38f224ffd4c1269ac8938b93bf3687c98a868e62dfdc664bd6782 not found: ID does not exist"
Jul 10 19:02:54.476020 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.475948    2558 scope.go:117] "RemoveContainer" containerID="f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7"
Jul 10 19:02:54.476375 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:02:54.476353    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7\": container with ID starting with f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7 not found: ID does not exist" containerID="f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7"
Jul 10 19:02:54.476476 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.476378    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7"} err="failed to get container status \"f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7\": rpc error: code = NotFound desc = could not find container \"f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7\": container with ID starting with f735bea9b45a4bf7c142f0926a0138bf734b137e28f410d5fa680f7bb6f73de7 not found: ID does not exist"
Jul 10 19:02:54.476476 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.476393    2558 scope.go:117] "RemoveContainer" containerID="6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39"
Jul 10 19:02:54.476629 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:02:54.476613    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39\": container with ID starting with 6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39 not found: ID does not exist" containerID="6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39"
Jul 10 19:02:54.476686 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.476631    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39"} err="failed to get container status \"6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39\": rpc error: code = NotFound desc = could not find container \"6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39\": container with ID starting with 6806e6d499089776461485c672d956c7af4d9c436876a746a8fc980de33ccd39 not found: ID does not exist"
Jul 10 19:02:54.477116 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:54.477099    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5c8676b6cb-h87kf"]
Jul 10 19:02:55.257215 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:55.257184    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" path="/var/lib/kubelet/pods/7f203187-df69-4531-ae76-ac5a3449a8be/volumes"
Jul 10 19:02:59.409963 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:02:59.409912    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 19:03:09.410504 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:09.410417    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 19:03:19.410158 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:19.410114    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 10 19:03:29.410911 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:29.410877    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:03:35.389384 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.389351    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"]
Jul 10 19:03:35.389832 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.389640    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kube-rbac-proxy" containerID="cri-o://28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18" gracePeriod=30
Jul 10 19:03:35.389832 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.389645    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container" containerID="cri-o://a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb" gracePeriod=30
Jul 10 19:03:35.479711 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.479675    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"]
Jul 10 19:03:35.479960 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.479948    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy"
Jul 10 19:03:35.480016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.479962    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy"
Jul 10 19:03:35.480016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.479974    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container"
Jul 10 19:03:35.480016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.479980    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container"
Jul 10 19:03:35.480016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.479987    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="storage-initializer"
Jul 10 19:03:35.480016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.479994    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="storage-initializer"
Jul 10 19:03:35.480016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.480000    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-agent"
Jul 10 19:03:35.480016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.480005    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-agent"
Jul 10 19:03:35.480239 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.480056    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-agent"
Jul 10 19:03:35.480239 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.480063    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kserve-container"
Jul 10 19:03:35.480239 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.480069    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f203187-df69-4531-ae76-ac5a3449a8be" containerName="kube-rbac-proxy"
Jul 10 19:03:35.482325 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.482305    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.484636 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.484616    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 10 19:03:35.484760 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.484672    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:03:35.496217 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.496190    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"]
Jul 10 19:03:35.549873 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.549840    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ced3e868-280e-4f0c-a2da-a2450c3cc59e-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.549873 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.549877    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scczw\" (UniqueName: \"kubernetes.io/projected/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kube-api-access-scczw\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.550065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.549908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced3e868-280e-4f0c-a2da-a2450c3cc59e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.550065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.549977    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.595523 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.595491    2558 generic.go:356] "Generic (PLEG): container finished" podID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerID="28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18" exitCode=2
Jul 10 19:03:35.595672 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.595529    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerDied","Data":"28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18"}
Jul 10 19:03:35.650896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.650821    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced3e868-280e-4f0c-a2da-a2450c3cc59e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.650896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.650866    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.651098 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.650911    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ced3e868-280e-4f0c-a2da-a2450c3cc59e-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.651098 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.650942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scczw\" (UniqueName: \"kubernetes.io/projected/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kube-api-access-scczw\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.651285 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.651265    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.651557 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.651539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ced3e868-280e-4f0c-a2da-a2450c3cc59e-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.653233 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.653210    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced3e868-280e-4f0c-a2da-a2450c3cc59e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.660227 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.660206    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scczw\" (UniqueName: \"kubernetes.io/projected/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kube-api-access-scczw\") pod \"isvc-paddle-runtime-predictor-6f68b485dd-prxcg\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.791952 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.791914    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:35.933476 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:35.933403    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"]
Jul 10 19:03:35.938881 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:03:35.938856    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced3e868_280e_4f0c_a2da_a2450c3cc59e.slice/crio-4a7a3d898768a103809393f9e4afb81cf960c69779320b6fb5d2252efaccf97a WatchSource:0}: Error finding container 4a7a3d898768a103809393f9e4afb81cf960c69779320b6fb5d2252efaccf97a: Status 404 returned error can't find the container with id 4a7a3d898768a103809393f9e4afb81cf960c69779320b6fb5d2252efaccf97a
Jul 10 19:03:36.599929 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:36.599891    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerStarted","Data":"6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c"}
Jul 10 19:03:36.600311 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:36.599945    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerStarted","Data":"4a7a3d898768a103809393f9e4afb81cf960c69779320b6fb5d2252efaccf97a"}
Jul 10 19:03:38.133899 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.133869    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:03:38.166846 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.166774    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd7e8d76-1089-427b-b386-8b0ce820e596-isvc-paddle-kube-rbac-proxy-sar-config\") pod \"bd7e8d76-1089-427b-b386-8b0ce820e596\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") "
Jul 10 19:03:38.166846 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.166807    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd7e8d76-1089-427b-b386-8b0ce820e596-proxy-tls\") pod \"bd7e8d76-1089-427b-b386-8b0ce820e596\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") "
Jul 10 19:03:38.167055 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.166850    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbzml\" (UniqueName: \"kubernetes.io/projected/bd7e8d76-1089-427b-b386-8b0ce820e596-kube-api-access-cbzml\") pod \"bd7e8d76-1089-427b-b386-8b0ce820e596\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") "
Jul 10 19:03:38.167055 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.166875    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7e8d76-1089-427b-b386-8b0ce820e596-kserve-provision-location\") pod \"bd7e8d76-1089-427b-b386-8b0ce820e596\" (UID: \"bd7e8d76-1089-427b-b386-8b0ce820e596\") "
Jul 10 19:03:38.167242 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.167218    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7e8d76-1089-427b-b386-8b0ce820e596-isvc-paddle-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-kube-rbac-proxy-sar-config") pod "bd7e8d76-1089-427b-b386-8b0ce820e596" (UID: "bd7e8d76-1089-427b-b386-8b0ce820e596"). InnerVolumeSpecName "isvc-paddle-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:03:38.169012 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.168988    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7e8d76-1089-427b-b386-8b0ce820e596-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd7e8d76-1089-427b-b386-8b0ce820e596" (UID: "bd7e8d76-1089-427b-b386-8b0ce820e596"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:03:38.169133 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.169030    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7e8d76-1089-427b-b386-8b0ce820e596-kube-api-access-cbzml" (OuterVolumeSpecName: "kube-api-access-cbzml") pod "bd7e8d76-1089-427b-b386-8b0ce820e596" (UID: "bd7e8d76-1089-427b-b386-8b0ce820e596"). InnerVolumeSpecName "kube-api-access-cbzml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:03:38.179194 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.179170    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd7e8d76-1089-427b-b386-8b0ce820e596-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd7e8d76-1089-427b-b386-8b0ce820e596" (UID: "bd7e8d76-1089-427b-b386-8b0ce820e596"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:03:38.268020 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.267985    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd7e8d76-1089-427b-b386-8b0ce820e596-isvc-paddle-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:03:38.268020 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.268013    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd7e8d76-1089-427b-b386-8b0ce820e596-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:03:38.268020 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.268025    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbzml\" (UniqueName: \"kubernetes.io/projected/bd7e8d76-1089-427b-b386-8b0ce820e596-kube-api-access-cbzml\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:03:38.268243 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.268034    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd7e8d76-1089-427b-b386-8b0ce820e596-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:03:38.608672 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.608639    2558 generic.go:356] "Generic (PLEG): container finished" podID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerID="a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb" exitCode=0
Jul 10 19:03:38.608897 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.608691    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"
Jul 10 19:03:38.608897 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.608703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerDied","Data":"a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb"}
Jul 10 19:03:38.608897 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.608788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm" event={"ID":"bd7e8d76-1089-427b-b386-8b0ce820e596","Type":"ContainerDied","Data":"a0ca70d898e2f2dbc984719b629e2895cdbe981f7e67b10273d6cb4bbaadaea0"}
Jul 10 19:03:38.608897 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.608827    2558 scope.go:117] "RemoveContainer" containerID="28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18"
Jul 10 19:03:38.616927 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.616911    2558 scope.go:117] "RemoveContainer" containerID="a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb"
Jul 10 19:03:38.624388 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.624369    2558 scope.go:117] "RemoveContainer" containerID="b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea"
Jul 10 19:03:38.631095 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.631081    2558 scope.go:117] "RemoveContainer" containerID="28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18"
Jul 10 19:03:38.631350 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:03:38.631329    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18\": container with ID starting with 28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18 not found: ID does not exist" containerID="28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18"
Jul 10 19:03:38.631425 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.631362    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18"} err="failed to get container status \"28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18\": rpc error: code = NotFound desc = could not find container \"28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18\": container with ID starting with 28e73ac51c88dbac63eca6824d1a8992fffb008788af9b669d94045d294e1e18 not found: ID does not exist"
Jul 10 19:03:38.631425 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.631387    2558 scope.go:117] "RemoveContainer" containerID="a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb"
Jul 10 19:03:38.631622 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:03:38.631605    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb\": container with ID starting with a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb not found: ID does not exist" containerID="a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb"
Jul 10 19:03:38.631889 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.631628    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb"} err="failed to get container status \"a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb\": rpc error: code = NotFound desc = could not find container \"a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb\": container with ID starting with a794b433c01b853439eb6d8f20f98376443100e1eae80bdcf602ab85221034eb not found: ID does not exist"
Jul 10 19:03:38.631889 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.631641    2558 scope.go:117] "RemoveContainer" containerID="b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea"
Jul 10 19:03:38.631889 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:03:38.631864    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea\": container with ID starting with b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea not found: ID does not exist" containerID="b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea"
Jul 10 19:03:38.631889 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.631881    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea"} err="failed to get container status \"b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea\": rpc error: code = NotFound desc = could not find container \"b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea\": container with ID starting with b78dbc71c8bcd2b3cc537986174acfa171c2a80461d85f4a8cd4cea60190c6ea not found: ID does not exist"
Jul 10 19:03:38.637317 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.637293    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"]
Jul 10 19:03:38.641656 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:38.641633    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-574f94bdcb-h24rm"]
Jul 10 19:03:39.256352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:39.256319    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" path="/var/lib/kubelet/pods/bd7e8d76-1089-427b-b386-8b0ce820e596/volumes"
Jul 10 19:03:41.622326 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:41.622287    2558 generic.go:356] "Generic (PLEG): container finished" podID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerID="6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c" exitCode=0
Jul 10 19:03:41.622696 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:41.622343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerDied","Data":"6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c"}
Jul 10 19:03:42.628039 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:42.628001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerStarted","Data":"0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8"}
Jul 10 19:03:42.628504 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:42.628063    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerStarted","Data":"3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7"}
Jul 10 19:03:42.628504 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:42.628326    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:42.628504 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:42.628460    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:42.629802 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:42.629775    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:03:42.650716 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:42.650664    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podStartSLOduration=2.886328611 podStartE2EDuration="7.650647325s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.764318714s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:03:42.649994551 +0000 UTC m=+1467.911659346" watchObservedRunningTime="2026-07-10 19:03:42.650647325 +0000 UTC m=+1467.912312100"
Jul 10 19:03:43.631937 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:43.631905    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:03:48.636072 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:48.636041    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:03:48.636613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:48.636586    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:03:58.637463 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:03:58.637423    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:04:08.636640 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:08.636595    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:04:15.258529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:15.258501    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:04:15.260207 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:15.260183    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:04:18.636839 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:18.636800    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:04:28.637606 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:28.637575    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:04:37.021997 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.021964    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"]
Jul 10 19:04:37.022437 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.022184    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" containerID="cri-o://3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7" gracePeriod=30
Jul 10 19:04:37.022437 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.022235    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kube-rbac-proxy" containerID="cri-o://0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8" gracePeriod=30
Jul 10 19:04:37.200563 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200534    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"]
Jul 10 19:04:37.200841 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200828    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="storage-initializer"
Jul 10 19:04:37.200891 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200843    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="storage-initializer"
Jul 10 19:04:37.200891 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200852    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container"
Jul 10 19:04:37.200891 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200858    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container"
Jul 10 19:04:37.200891 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200872    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kube-rbac-proxy"
Jul 10 19:04:37.200891 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200880    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kube-rbac-proxy"
Jul 10 19:04:37.201044 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200930    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kube-rbac-proxy"
Jul 10 19:04:37.201044 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.200937    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd7e8d76-1089-427b-b386-8b0ce820e596" containerName="kserve-container"
Jul 10 19:04:37.202827 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.202813    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.207696 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.207673    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 10 19:04:37.207696 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.207695    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 10 19:04:37.217947 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.217911    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"]
Jul 10 19:04:37.308573 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.308484    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11c5714c-c530-4d26-acb0-e32e37cc121d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.308573 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.308521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l64sj\" (UniqueName: \"kubernetes.io/projected/11c5714c-c530-4d26-acb0-e32e37cc121d-kube-api-access-l64sj\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.308573 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.308551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11c5714c-c530-4d26-acb0-e32e37cc121d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.308917 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.308599    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11c5714c-c530-4d26-acb0-e32e37cc121d-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.409138 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.409094    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11c5714c-c530-4d26-acb0-e32e37cc121d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.409138 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.409135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l64sj\" (UniqueName: \"kubernetes.io/projected/11c5714c-c530-4d26-acb0-e32e37cc121d-kube-api-access-l64sj\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.409394 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.409162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11c5714c-c530-4d26-acb0-e32e37cc121d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.409394 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.409196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11c5714c-c530-4d26-acb0-e32e37cc121d-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.409570 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.409547    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11c5714c-c530-4d26-acb0-e32e37cc121d-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.409943 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.409920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11c5714c-c530-4d26-acb0-e32e37cc121d-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.411654 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.411631    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11c5714c-c530-4d26-acb0-e32e37cc121d-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.418065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.418032    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l64sj\" (UniqueName: \"kubernetes.io/projected/11c5714c-c530-4d26-acb0-e32e37cc121d-kube-api-access-l64sj\") pod \"isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.514018 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.513972    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:37.656407 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.656374    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"]
Jul 10 19:04:37.659256 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:04:37.659229    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c5714c_c530_4d26_acb0_e32e37cc121d.slice/crio-f1a7e0f7a80188f607c61a00c6401264e3ac24f6b1133e484bc9620af215e7d4 WatchSource:0}: Error finding container f1a7e0f7a80188f607c61a00c6401264e3ac24f6b1133e484bc9620af215e7d4: Status 404 returned error can't find the container with id f1a7e0f7a80188f607c61a00c6401264e3ac24f6b1133e484bc9620af215e7d4
Jul 10 19:04:37.827922 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.827832    2558 generic.go:356] "Generic (PLEG): container finished" podID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerID="0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8" exitCode=2
Jul 10 19:04:37.828077 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.827911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerDied","Data":"0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8"}
Jul 10 19:04:37.829460 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.829437    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerStarted","Data":"a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b"}
Jul 10 19:04:37.829583 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:37.829484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerStarted","Data":"f1a7e0f7a80188f607c61a00c6401264e3ac24f6b1133e484bc9620af215e7d4"}
Jul 10 19:04:38.632296 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:38.632254    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 10 19:04:38.636553 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:38.636525    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 10 19:04:39.765763 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.765726    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:04:39.838762 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.838717    2558 generic.go:356] "Generic (PLEG): container finished" podID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerID="3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7" exitCode=0
Jul 10 19:04:39.838940 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.838773    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerDied","Data":"3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7"}
Jul 10 19:04:39.838940 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.838794    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"
Jul 10 19:04:39.838940 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.838830    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg" event={"ID":"ced3e868-280e-4f0c-a2da-a2450c3cc59e","Type":"ContainerDied","Data":"4a7a3d898768a103809393f9e4afb81cf960c69779320b6fb5d2252efaccf97a"}
Jul 10 19:04:39.838940 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.838858    2558 scope.go:117] "RemoveContainer" containerID="0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8"
Jul 10 19:04:39.846417 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.846399    2558 scope.go:117] "RemoveContainer" containerID="3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7"
Jul 10 19:04:39.853262 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.853242    2558 scope.go:117] "RemoveContainer" containerID="6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c"
Jul 10 19:04:39.859928 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.859910    2558 scope.go:117] "RemoveContainer" containerID="0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8"
Jul 10 19:04:39.860163 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:04:39.860144    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8\": container with ID starting with 0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8 not found: ID does not exist" containerID="0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8"
Jul 10 19:04:39.860212 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.860173    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8"} err="failed to get container status \"0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8\": rpc error: code = NotFound desc = could not find container \"0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8\": container with ID starting with 0a3b9526b5158c17a389b14c9e3bd03e58890afad4b8cbb622b34457da0ae5e8 not found: ID does not exist"
Jul 10 19:04:39.860212 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.860193    2558 scope.go:117] "RemoveContainer" containerID="3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7"
Jul 10 19:04:39.860416 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:04:39.860397    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7\": container with ID starting with 3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7 not found: ID does not exist" containerID="3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7"
Jul 10 19:04:39.860453 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.860423    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7"} err="failed to get container status \"3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7\": rpc error: code = NotFound desc = could not find container \"3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7\": container with ID starting with 3d5d901db8c601c0a014ae8e155df940abdf73693f3594772617f802446263a7 not found: ID does not exist"
Jul 10 19:04:39.860453 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.860440    2558 scope.go:117] "RemoveContainer" containerID="6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c"
Jul 10 19:04:39.860669 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:04:39.860653    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c\": container with ID starting with 6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c not found: ID does not exist" containerID="6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c"
Jul 10 19:04:39.860717 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.860674    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c"} err="failed to get container status \"6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c\": rpc error: code = NotFound desc = could not find container \"6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c\": container with ID starting with 6c712c8b6bb2aef7c9ba8793d9fa96f678076402cda99a43559988498766cd5c not found: ID does not exist"
Jul 10 19:04:39.931605 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.931571    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scczw\" (UniqueName: \"kubernetes.io/projected/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kube-api-access-scczw\") pod \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") "
Jul 10 19:04:39.931774 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.931623    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ced3e868-280e-4f0c-a2da-a2450c3cc59e-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") pod \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") "
Jul 10 19:04:39.931774 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.931655    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced3e868-280e-4f0c-a2da-a2450c3cc59e-proxy-tls\") pod \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") "
Jul 10 19:04:39.931774 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.931689    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kserve-provision-location\") pod \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\" (UID: \"ced3e868-280e-4f0c-a2da-a2450c3cc59e\") "
Jul 10 19:04:39.932036 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.932006    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced3e868-280e-4f0c-a2da-a2450c3cc59e-isvc-paddle-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-runtime-kube-rbac-proxy-sar-config") pod "ced3e868-280e-4f0c-a2da-a2450c3cc59e" (UID: "ced3e868-280e-4f0c-a2da-a2450c3cc59e"). InnerVolumeSpecName "isvc-paddle-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:04:39.933671 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.933640    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ced3e868-280e-4f0c-a2da-a2450c3cc59e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ced3e868-280e-4f0c-a2da-a2450c3cc59e" (UID: "ced3e868-280e-4f0c-a2da-a2450c3cc59e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:04:39.933773 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.933688    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kube-api-access-scczw" (OuterVolumeSpecName: "kube-api-access-scczw") pod "ced3e868-280e-4f0c-a2da-a2450c3cc59e" (UID: "ced3e868-280e-4f0c-a2da-a2450c3cc59e"). InnerVolumeSpecName "kube-api-access-scczw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:04:39.941882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:39.941856    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ced3e868-280e-4f0c-a2da-a2450c3cc59e" (UID: "ced3e868-280e-4f0c-a2da-a2450c3cc59e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:04:40.032697 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:40.032664    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scczw\" (UniqueName: \"kubernetes.io/projected/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kube-api-access-scczw\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:04:40.032697 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:40.032691    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ced3e868-280e-4f0c-a2da-a2450c3cc59e-isvc-paddle-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:04:40.032697 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:40.032702    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ced3e868-280e-4f0c-a2da-a2450c3cc59e-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:04:40.032954 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:40.032713    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ced3e868-280e-4f0c-a2da-a2450c3cc59e-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:04:40.161352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:40.161319    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"]
Jul 10 19:04:40.169225 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:40.169199    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-6f68b485dd-prxcg"]
Jul 10 19:04:41.256629 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:41.256589    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" path="/var/lib/kubelet/pods/ced3e868-280e-4f0c-a2da-a2450c3cc59e/volumes"
Jul 10 19:04:42.853604 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:42.853566    2558 generic.go:356] "Generic (PLEG): container finished" podID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerID="a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b" exitCode=0
Jul 10 19:04:42.854097 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:42.853619    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerDied","Data":"a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b"}
Jul 10 19:04:43.859647 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:43.859562    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerStarted","Data":"0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a"}
Jul 10 19:04:43.859647 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:43.859622    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerStarted","Data":"9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b"}
Jul 10 19:04:43.860085 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:43.859857    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:43.860085 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:43.860017    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:43.861045 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:43.861016    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 10 19:04:43.883737 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:43.883686    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podStartSLOduration=2.247559407 podStartE2EDuration="6.883671071s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.636111664s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:43.88246632 +0000 UTC m=+1529.144131090" watchObservedRunningTime="2026-07-10 19:04:43.883671071 +0000 UTC m=+1529.145335845"
Jul 10 19:04:44.863603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:44.863562    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 10 19:04:49.868182 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:49.868150    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:04:49.868666 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:49.868639    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 10 19:04:59.869306 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:04:59.869264    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 10 19:05:09.869413 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:09.869372    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 10 19:05:19.868968 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:19.868926    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 10 19:05:29.869365 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:29.869334    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:05:38.952861 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:38.952826    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"]
Jul 10 19:05:38.953328 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:38.953078    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" containerID="cri-o://9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b" gracePeriod=30
Jul 10 19:05:38.953328 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:38.953100    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kube-rbac-proxy" containerID="cri-o://0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a" gracePeriod=30
Jul 10 19:05:39.144309 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144273    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"]
Jul 10 19:05:39.144546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144534    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kube-rbac-proxy"
Jul 10 19:05:39.144546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144547    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kube-rbac-proxy"
Jul 10 19:05:39.144629 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144559    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="storage-initializer"
Jul 10 19:05:39.144629 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144566    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="storage-initializer"
Jul 10 19:05:39.144629 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144572    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container"
Jul 10 19:05:39.144629 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144579    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container"
Jul 10 19:05:39.144629 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144624    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kserve-container"
Jul 10 19:05:39.144809 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.144631    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ced3e868-280e-4f0c-a2da-a2450c3cc59e" containerName="kube-rbac-proxy"
Jul 10 19:05:39.147461 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.147445    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.153125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.153083    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 10 19:05:39.153291 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.153181    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 10 19:05:39.168777 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.168732    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"]
Jul 10 19:05:39.240617 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.240532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/ab7d08f2-4e64-4a77-b932-d52b26f37115-kube-api-access-fvnxv\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.240617 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.240607    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7d08f2-4e64-4a77-b932-d52b26f37115-proxy-tls\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.240876 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.240680    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7d08f2-4e64-4a77-b932-d52b26f37115-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.240876 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.240722    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7d08f2-4e64-4a77-b932-d52b26f37115-kserve-provision-location\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.341953 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.341922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7d08f2-4e64-4a77-b932-d52b26f37115-proxy-tls\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.342136 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.341966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7d08f2-4e64-4a77-b932-d52b26f37115-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.342136 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.341986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7d08f2-4e64-4a77-b932-d52b26f37115-kserve-provision-location\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.342136 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.342006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/ab7d08f2-4e64-4a77-b932-d52b26f37115-kube-api-access-fvnxv\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.342421 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.342397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7d08f2-4e64-4a77-b932-d52b26f37115-kserve-provision-location\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.342698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.342680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7d08f2-4e64-4a77-b932-d52b26f37115-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.344463 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.344441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7d08f2-4e64-4a77-b932-d52b26f37115-proxy-tls\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.357696 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.357667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/ab7d08f2-4e64-4a77-b932-d52b26f37115-kube-api-access-fvnxv\") pod \"isvc-pmml-predictor-667774f679-s6k7z\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.457324 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.457289    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:39.589652 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.589630    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"]
Jul 10 19:05:39.591618 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:05:39.591583    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7d08f2_4e64_4a77_b932_d52b26f37115.slice/crio-269d3118c860c3550e1269b36b03685e5573d14c94cc26c4de1f7a2405374e18 WatchSource:0}: Error finding container 269d3118c860c3550e1269b36b03685e5573d14c94cc26c4de1f7a2405374e18: Status 404 returned error can't find the container with id 269d3118c860c3550e1269b36b03685e5573d14c94cc26c4de1f7a2405374e18
Jul 10 19:05:39.864143 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.864063    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 10 19:05:39.869396 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:39.869366    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 10 19:05:40.057938 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:40.057902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerStarted","Data":"89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3"}
Jul 10 19:05:40.058389 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:40.057976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerStarted","Data":"269d3118c860c3550e1269b36b03685e5573d14c94cc26c4de1f7a2405374e18"}
Jul 10 19:05:40.060141 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:40.060116    2558 generic.go:356] "Generic (PLEG): container finished" podID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerID="0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a" exitCode=2
Jul 10 19:05:40.060263 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:40.060187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerDied","Data":"0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a"}
Jul 10 19:05:41.501049 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.501028    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:05:41.558869 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.558841    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11c5714c-c530-4d26-acb0-e32e37cc121d-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") pod \"11c5714c-c530-4d26-acb0-e32e37cc121d\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") "
Jul 10 19:05:41.559043 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.558912    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l64sj\" (UniqueName: \"kubernetes.io/projected/11c5714c-c530-4d26-acb0-e32e37cc121d-kube-api-access-l64sj\") pod \"11c5714c-c530-4d26-acb0-e32e37cc121d\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") "
Jul 10 19:05:41.559043 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.558956    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11c5714c-c530-4d26-acb0-e32e37cc121d-kserve-provision-location\") pod \"11c5714c-c530-4d26-acb0-e32e37cc121d\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") "
Jul 10 19:05:41.559043 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.559006    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11c5714c-c530-4d26-acb0-e32e37cc121d-proxy-tls\") pod \"11c5714c-c530-4d26-acb0-e32e37cc121d\" (UID: \"11c5714c-c530-4d26-acb0-e32e37cc121d\") "
Jul 10 19:05:41.559260 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.559242    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c5714c-c530-4d26-acb0-e32e37cc121d-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config") pod "11c5714c-c530-4d26-acb0-e32e37cc121d" (UID: "11c5714c-c530-4d26-acb0-e32e37cc121d"). InnerVolumeSpecName "isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:05:41.561064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.561031    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c5714c-c530-4d26-acb0-e32e37cc121d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11c5714c-c530-4d26-acb0-e32e37cc121d" (UID: "11c5714c-c530-4d26-acb0-e32e37cc121d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:05:41.561208 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.561080    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c5714c-c530-4d26-acb0-e32e37cc121d-kube-api-access-l64sj" (OuterVolumeSpecName: "kube-api-access-l64sj") pod "11c5714c-c530-4d26-acb0-e32e37cc121d" (UID: "11c5714c-c530-4d26-acb0-e32e37cc121d"). InnerVolumeSpecName "kube-api-access-l64sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:05:41.573859 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.573803    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c5714c-c530-4d26-acb0-e32e37cc121d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "11c5714c-c530-4d26-acb0-e32e37cc121d" (UID: "11c5714c-c530-4d26-acb0-e32e37cc121d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:05:41.660176 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.660144    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l64sj\" (UniqueName: \"kubernetes.io/projected/11c5714c-c530-4d26-acb0-e32e37cc121d-kube-api-access-l64sj\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:05:41.660176 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.660173    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/11c5714c-c530-4d26-acb0-e32e37cc121d-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:05:41.660176 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.660183    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11c5714c-c530-4d26-acb0-e32e37cc121d-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:05:41.660396 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:41.660192    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11c5714c-c530-4d26-acb0-e32e37cc121d-isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:05:42.069695 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.069663    2558 generic.go:356] "Generic (PLEG): container finished" podID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerID="9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b" exitCode=0
Jul 10 19:05:42.069882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.069716    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"
Jul 10 19:05:42.069882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.069733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerDied","Data":"9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b"}
Jul 10 19:05:42.069882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.069806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn" event={"ID":"11c5714c-c530-4d26-acb0-e32e37cc121d","Type":"ContainerDied","Data":"f1a7e0f7a80188f607c61a00c6401264e3ac24f6b1133e484bc9620af215e7d4"}
Jul 10 19:05:42.069882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.069832    2558 scope.go:117] "RemoveContainer" containerID="0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a"
Jul 10 19:05:42.078425 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.078408    2558 scope.go:117] "RemoveContainer" containerID="9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b"
Jul 10 19:05:42.085354 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.085339    2558 scope.go:117] "RemoveContainer" containerID="a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b"
Jul 10 19:05:42.092279 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.092249    2558 scope.go:117] "RemoveContainer" containerID="0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a"
Jul 10 19:05:42.092627 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:05:42.092605    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a\": container with ID starting with 0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a not found: ID does not exist" containerID="0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a"
Jul 10 19:05:42.092702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.092634    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a"} err="failed to get container status \"0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a\": rpc error: code = NotFound desc = could not find container \"0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a\": container with ID starting with 0dc289583fe3aaa86f1349913fa0ec1c769cb39f1506c75a8a382e992718eb8a not found: ID does not exist"
Jul 10 19:05:42.092702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.092651    2558 scope.go:117] "RemoveContainer" containerID="9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b"
Jul 10 19:05:42.092979 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:05:42.092962    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b\": container with ID starting with 9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b not found: ID does not exist" containerID="9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b"
Jul 10 19:05:42.093034 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.092983    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b"} err="failed to get container status \"9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b\": rpc error: code = NotFound desc = could not find container \"9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b\": container with ID starting with 9069084f1a5306114e7b3f0abd63fd86c1d40b195998d8db36318a1566539a5b not found: ID does not exist"
Jul 10 19:05:42.093034 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.093001    2558 scope.go:117] "RemoveContainer" containerID="a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b"
Jul 10 19:05:42.093229 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:05:42.093211    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b\": container with ID starting with a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b not found: ID does not exist" containerID="a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b"
Jul 10 19:05:42.093285 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.093235    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b"} err="failed to get container status \"a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b\": rpc error: code = NotFound desc = could not find container \"a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b\": container with ID starting with a37e674e0ab168b93f69ba6df0f593c8b99a9216fd39b0985ad8449f40885b3b not found: ID does not exist"
Jul 10 19:05:42.093816 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.093800    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"]
Jul 10 19:05:42.099289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:42.099270    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-58879dd4d4-slznn"]
Jul 10 19:05:43.256306 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:43.256274    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" path="/var/lib/kubelet/pods/11c5714c-c530-4d26-acb0-e32e37cc121d/volumes"
Jul 10 19:05:44.082679 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:44.082646    2558 generic.go:356] "Generic (PLEG): container finished" podID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerID="89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3" exitCode=0
Jul 10 19:05:44.082900 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:44.082693    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerDied","Data":"89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3"}
Jul 10 19:05:52.118958 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:52.118924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerStarted","Data":"24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7"}
Jul 10 19:05:52.119369 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:52.119009    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerStarted","Data":"8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099"}
Jul 10 19:05:52.119369 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:52.119270    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:52.119483 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:52.119401    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:52.120683 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:52.120655    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:05:52.145600 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:52.145557    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podStartSLOduration=2.007823539 podStartE2EDuration="13.145546407s" totalImagesPullingTime="7.191499784s" totalInitContainerRuntime="3.946223084s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:52.145309309 +0000 UTC m=+1597.406974082" watchObservedRunningTime="2026-07-10 19:05:52.145546407 +0000 UTC m=+1597.407211182"
Jul 10 19:05:53.123673 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:53.123634    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:05:58.128415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:58.128387    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:05:58.128876 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:05:58.128851    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:06:08.129602 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:06:08.129517    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:06:18.129702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:06:18.129658    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:06:28.129536 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:06:28.129496    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:06:38.129099 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:06:38.129061    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:06:48.129243 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:06:48.129208    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:06:58.129441 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:06:58.129399    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 10 19:07:08.129832 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:08.129805    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:07:10.279182 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.279143    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"]
Jul 10 19:07:10.279619 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.279393    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container" containerID="cri-o://8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099" gracePeriod=30
Jul 10 19:07:10.279619 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.279448    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kube-rbac-proxy" containerID="cri-o://24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7" gracePeriod=30
Jul 10 19:07:10.288970 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.288945    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"]
Jul 10 19:07:10.289366 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289351    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kube-rbac-proxy"
Jul 10 19:07:10.289412 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289371    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kube-rbac-proxy"
Jul 10 19:07:10.289412 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289397    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container"
Jul 10 19:07:10.289412 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289407    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container"
Jul 10 19:07:10.289578 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289420    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="storage-initializer"
Jul 10 19:07:10.289578 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289430    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="storage-initializer"
Jul 10 19:07:10.289578 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289510    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kserve-container"
Jul 10 19:07:10.289578 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.289524    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c5714c-c530-4d26-acb0-e32e37cc121d" containerName="kube-rbac-proxy"
Jul 10 19:07:10.292733 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.292714    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.297073 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.297055    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:07:10.300066 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.300045    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 10 19:07:10.310916 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.310893    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"]
Jul 10 19:07:10.400917 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.400889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.401060 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.400930    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.401060 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.400960    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7r5\" (UniqueName: \"kubernetes.io/projected/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kube-api-access-dj7r5\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.401060 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.401000    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.502211 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.502180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7r5\" (UniqueName: \"kubernetes.io/projected/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kube-api-access-dj7r5\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.502407 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.502224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.502407 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.502271    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.502407 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.502318    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.502407 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:07:10.502382    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 10 19:07:10.502622 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:07:10.502457    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls podName:2c45ce15-cd8f-4a6d-8962-f858cdfd81eb nodeName:}" failed. No retries permitted until 2026-07-10 19:07:11.002435292 +0000 UTC m=+1676.264100063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls") pod "isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" (UID: "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Jul 10 19:07:10.502773 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.502726    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.503123 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.503107    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:10.518071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:10.518050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7r5\" (UniqueName: \"kubernetes.io/projected/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kube-api-access-dj7r5\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:11.005104 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:11.005073    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:11.007575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:11.007540    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls\") pod \"isvc-pmml-runtime-predictor-56f7c95d45-b7mxp\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:11.202224 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:11.202178    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:11.336428 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:11.336391    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"]
Jul 10 19:07:11.339519 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:07:11.339494    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c45ce15_cd8f_4a6d_8962_f858cdfd81eb.slice/crio-68b87159d1b5488b9ba0db76a03b5f2d6884c69d746c7ea34a3b6224a8a7d9df WatchSource:0}: Error finding container 68b87159d1b5488b9ba0db76a03b5f2d6884c69d746c7ea34a3b6224a8a7d9df: Status 404 returned error can't find the container with id 68b87159d1b5488b9ba0db76a03b5f2d6884c69d746c7ea34a3b6224a8a7d9df
Jul 10 19:07:11.393672 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:11.393642    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerStarted","Data":"68b87159d1b5488b9ba0db76a03b5f2d6884c69d746c7ea34a3b6224a8a7d9df"}
Jul 10 19:07:11.395719 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:11.395698    2558 generic.go:356] "Generic (PLEG): container finished" podID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerID="24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7" exitCode=2
Jul 10 19:07:11.395822 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:11.395734    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerDied","Data":"24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7"}
Jul 10 19:07:12.401353 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:12.401318    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerStarted","Data":"4cba5ec85dc644e2f2cfae296f4d7be8dc4d1eed87c52a6b7bf578610fa74786"}
Jul 10 19:07:13.124027 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.123982    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 10 19:07:13.727024 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.727000    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:07:13.830270 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.830206    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7d08f2-4e64-4a77-b932-d52b26f37115-kserve-provision-location\") pod \"ab7d08f2-4e64-4a77-b932-d52b26f37115\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") "
Jul 10 19:07:13.830270 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.830256    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7d08f2-4e64-4a77-b932-d52b26f37115-proxy-tls\") pod \"ab7d08f2-4e64-4a77-b932-d52b26f37115\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") "
Jul 10 19:07:13.830443 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.830294    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/ab7d08f2-4e64-4a77-b932-d52b26f37115-kube-api-access-fvnxv\") pod \"ab7d08f2-4e64-4a77-b932-d52b26f37115\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") "
Jul 10 19:07:13.830443 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.830341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7d08f2-4e64-4a77-b932-d52b26f37115-isvc-pmml-kube-rbac-proxy-sar-config\") pod \"ab7d08f2-4e64-4a77-b932-d52b26f37115\" (UID: \"ab7d08f2-4e64-4a77-b932-d52b26f37115\") "
Jul 10 19:07:13.830570 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.830549    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7d08f2-4e64-4a77-b932-d52b26f37115-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab7d08f2-4e64-4a77-b932-d52b26f37115" (UID: "ab7d08f2-4e64-4a77-b932-d52b26f37115"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:07:13.830770 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.830725    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7d08f2-4e64-4a77-b932-d52b26f37115-isvc-pmml-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-kube-rbac-proxy-sar-config") pod "ab7d08f2-4e64-4a77-b932-d52b26f37115" (UID: "ab7d08f2-4e64-4a77-b932-d52b26f37115"). InnerVolumeSpecName "isvc-pmml-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:07:13.832303 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.832277    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7d08f2-4e64-4a77-b932-d52b26f37115-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab7d08f2-4e64-4a77-b932-d52b26f37115" (UID: "ab7d08f2-4e64-4a77-b932-d52b26f37115"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:07:13.832401 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.832328    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7d08f2-4e64-4a77-b932-d52b26f37115-kube-api-access-fvnxv" (OuterVolumeSpecName: "kube-api-access-fvnxv") pod "ab7d08f2-4e64-4a77-b932-d52b26f37115" (UID: "ab7d08f2-4e64-4a77-b932-d52b26f37115"). InnerVolumeSpecName "kube-api-access-fvnxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:07:13.931341 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.931310    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab7d08f2-4e64-4a77-b932-d52b26f37115-isvc-pmml-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:07:13.931341 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.931336    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab7d08f2-4e64-4a77-b932-d52b26f37115-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:07:13.931511 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.931350    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab7d08f2-4e64-4a77-b932-d52b26f37115-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:07:13.931511 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:13.931363    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/ab7d08f2-4e64-4a77-b932-d52b26f37115-kube-api-access-fvnxv\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:07:14.410173 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.410142    2558 generic.go:356] "Generic (PLEG): container finished" podID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerID="8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099" exitCode=0
Jul 10 19:07:14.410339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.410184    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerDied","Data":"8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099"}
Jul 10 19:07:14.410339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.410205    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"
Jul 10 19:07:14.410339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.410228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z" event={"ID":"ab7d08f2-4e64-4a77-b932-d52b26f37115","Type":"ContainerDied","Data":"269d3118c860c3550e1269b36b03685e5573d14c94cc26c4de1f7a2405374e18"}
Jul 10 19:07:14.410339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.410253    2558 scope.go:117] "RemoveContainer" containerID="24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7"
Jul 10 19:07:14.418352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.418306    2558 scope.go:117] "RemoveContainer" containerID="8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099"
Jul 10 19:07:14.425837 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.425820    2558 scope.go:117] "RemoveContainer" containerID="89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3"
Jul 10 19:07:14.432295 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.432278    2558 scope.go:117] "RemoveContainer" containerID="24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7"
Jul 10 19:07:14.432524 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:07:14.432507    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7\": container with ID starting with 24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7 not found: ID does not exist" containerID="24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7"
Jul 10 19:07:14.432571 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.432532    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7"} err="failed to get container status \"24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7\": rpc error: code = NotFound desc = could not find container \"24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7\": container with ID starting with 24ac3265dc7821553904b0d36da346cc958db0b9737f847660af7bbab13c55a7 not found: ID does not exist"
Jul 10 19:07:14.432571 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.432549    2558 scope.go:117] "RemoveContainer" containerID="8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099"
Jul 10 19:07:14.432824 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:07:14.432806    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099\": container with ID starting with 8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099 not found: ID does not exist" containerID="8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099"
Jul 10 19:07:14.432895 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.432831    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099"} err="failed to get container status \"8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099\": rpc error: code = NotFound desc = could not find container \"8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099\": container with ID starting with 8424a3f0f0858ea0edfe0c4aa8ea65f497823aae9c6f00c80789ed6363991099 not found: ID does not exist"
Jul 10 19:07:14.432895 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.432847    2558 scope.go:117] "RemoveContainer" containerID="89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3"
Jul 10 19:07:14.433258 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:07:14.433167    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3\": container with ID starting with 89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3 not found: ID does not exist" containerID="89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3"
Jul 10 19:07:14.433258 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.433195    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3"} err="failed to get container status \"89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3\": rpc error: code = NotFound desc = could not find container \"89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3\": container with ID starting with 89570ace141e6748a90abacc2d304905430e09513cea2f87415192b56cd118b3 not found: ID does not exist"
Jul 10 19:07:14.434965 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.434941    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"]
Jul 10 19:07:14.439630 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:14.439609    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-667774f679-s6k7z"]
Jul 10 19:07:15.256414 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:15.256333    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" path="/var/lib/kubelet/pods/ab7d08f2-4e64-4a77-b932-d52b26f37115/volumes"
Jul 10 19:07:15.414716 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:15.414690    2558 generic.go:356] "Generic (PLEG): container finished" podID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerID="4cba5ec85dc644e2f2cfae296f4d7be8dc4d1eed87c52a6b7bf578610fa74786" exitCode=0
Jul 10 19:07:15.414872 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:15.414765    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerDied","Data":"4cba5ec85dc644e2f2cfae296f4d7be8dc4d1eed87c52a6b7bf578610fa74786"}
Jul 10 19:07:16.419996 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:16.419963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerStarted","Data":"886337e2db54a14b526f9a86a881d81c3716d7eeccc42dd9a3456174d5b1da76"}
Jul 10 19:07:16.420378 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:16.420027    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerStarted","Data":"1dfd2bea3461d493c735f15759f9d4d7fdc37510052b88b209ee211f74c2530b"}
Jul 10 19:07:16.420378 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:16.420309    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:16.441592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:16.441540    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podStartSLOduration=2.8561100550000003 podStartE2EDuration="6.441522144s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585412089s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:16.440804074 +0000 UTC m=+1681.702468847" watchObservedRunningTime="2026-07-10 19:07:16.441522144 +0000 UTC m=+1681.703186918"
Jul 10 19:07:17.423404 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:17.423376    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:17.424615 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:17.424592    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:07:18.426471 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:18.426436    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:07:23.435609 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:23.435575    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:07:23.436613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:23.436406    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:07:33.437120 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:33.437079    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:07:43.437253 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:43.437156    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:07:53.436817 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:07:53.436780    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:08:03.437205 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:03.437168    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:08:13.436704 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:13.436666    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:08:23.436484 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:23.436447    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:08:33.437636 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:33.437604    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:08:41.371071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.371033    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"]
Jul 10 19:08:41.371479 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.371295    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" containerID="cri-o://1dfd2bea3461d493c735f15759f9d4d7fdc37510052b88b209ee211f74c2530b" gracePeriod=30
Jul 10 19:08:41.371479 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.371312    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kube-rbac-proxy" containerID="cri-o://886337e2db54a14b526f9a86a881d81c3716d7eeccc42dd9a3456174d5b1da76" gracePeriod=30
Jul 10 19:08:41.385589 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.385564    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"]
Jul 10 19:08:41.385909 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.385892    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="storage-initializer"
Jul 10 19:08:41.385998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.385912    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="storage-initializer"
Jul 10 19:08:41.385998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.385932    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kube-rbac-proxy"
Jul 10 19:08:41.385998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.385941    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kube-rbac-proxy"
Jul 10 19:08:41.385998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.385960    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container"
Jul 10 19:08:41.385998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.385970    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container"
Jul 10 19:08:41.386256 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.386047    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kube-rbac-proxy"
Jul 10 19:08:41.386256 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.386061    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7d08f2-4e64-4a77-b932-d52b26f37115" containerName="kserve-container"
Jul 10 19:08:41.388970 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.388952    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.391251 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.391232    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 10 19:08:41.391390 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.391374    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 10 19:08:41.399378 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.399356    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"]
Jul 10 19:08:41.458674 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.458647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d986c595-2a94-48fb-a851-1ffa7aab3389-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.458875 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.458686    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d986c595-2a94-48fb-a851-1ffa7aab3389-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.458875 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.458717    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d986c595-2a94-48fb-a851-1ffa7aab3389-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.458875 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.458827    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84bn\" (UniqueName: \"kubernetes.io/projected/d986c595-2a94-48fb-a851-1ffa7aab3389-kube-api-access-l84bn\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.559673 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.559639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d986c595-2a94-48fb-a851-1ffa7aab3389-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.559860 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.559686    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d986c595-2a94-48fb-a851-1ffa7aab3389-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.559860 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.559719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d986c595-2a94-48fb-a851-1ffa7aab3389-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.559860 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.559756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84bn\" (UniqueName: \"kubernetes.io/projected/d986c595-2a94-48fb-a851-1ffa7aab3389-kube-api-access-l84bn\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.560139 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.560115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d986c595-2a94-48fb-a851-1ffa7aab3389-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.560472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.560452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d986c595-2a94-48fb-a851-1ffa7aab3389-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.562065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.562042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d986c595-2a94-48fb-a851-1ffa7aab3389-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.568031 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.568012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84bn\" (UniqueName: \"kubernetes.io/projected/d986c595-2a94-48fb-a851-1ffa7aab3389-kube-api-access-l84bn\") pod \"isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.698494 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.698408    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:41.717979 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.717955    2558 generic.go:356] "Generic (PLEG): container finished" podID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerID="886337e2db54a14b526f9a86a881d81c3716d7eeccc42dd9a3456174d5b1da76" exitCode=2
Jul 10 19:08:41.718093 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.718048    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerDied","Data":"886337e2db54a14b526f9a86a881d81c3716d7eeccc42dd9a3456174d5b1da76"}
Jul 10 19:08:41.821640 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:41.821614    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"]
Jul 10 19:08:41.823893 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:08:41.823865    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd986c595_2a94_48fb_a851_1ffa7aab3389.slice/crio-16238047afb99c9886de1fd95a10805b1cfbfa7cdaac708820657e5011226099 WatchSource:0}: Error finding container 16238047afb99c9886de1fd95a10805b1cfbfa7cdaac708820657e5011226099: Status 404 returned error can't find the container with id 16238047afb99c9886de1fd95a10805b1cfbfa7cdaac708820657e5011226099
Jul 10 19:08:42.723705 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:42.723666    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerStarted","Data":"f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd"}
Jul 10 19:08:42.724093 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:42.723720    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerStarted","Data":"16238047afb99c9886de1fd95a10805b1cfbfa7cdaac708820657e5011226099"}
Jul 10 19:08:43.426919 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:43.426881    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Jul 10 19:08:43.437194 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:43.437166    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 10 19:08:44.732620 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.732590    2558 generic.go:356] "Generic (PLEG): container finished" podID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerID="1dfd2bea3461d493c735f15759f9d4d7fdc37510052b88b209ee211f74c2530b" exitCode=0
Jul 10 19:08:44.732991 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.732648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerDied","Data":"1dfd2bea3461d493c735f15759f9d4d7fdc37510052b88b209ee211f74c2530b"}
Jul 10 19:08:44.813728 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.813706    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:08:44.883800 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.883703    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls\") pod \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") "
Jul 10 19:08:44.883800 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.883767    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj7r5\" (UniqueName: \"kubernetes.io/projected/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kube-api-access-dj7r5\") pod \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") "
Jul 10 19:08:44.883800 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.883789    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kserve-provision-location\") pod \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") "
Jul 10 19:08:44.884063 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.883845    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") pod \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\" (UID: \"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb\") "
Jul 10 19:08:44.884188 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.884161    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" (UID: "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:08:44.884234 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.884212    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-isvc-pmml-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-runtime-kube-rbac-proxy-sar-config") pod "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" (UID: "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb"). InnerVolumeSpecName "isvc-pmml-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:08:44.885793 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.885772    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" (UID: "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:08:44.885876 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.885829    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kube-api-access-dj7r5" (OuterVolumeSpecName: "kube-api-access-dj7r5") pod "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" (UID: "2c45ce15-cd8f-4a6d-8962-f858cdfd81eb"). InnerVolumeSpecName "kube-api-access-dj7r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:08:44.984526 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.984503    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:08:44.984526 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.984523    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj7r5\" (UniqueName: \"kubernetes.io/projected/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kube-api-access-dj7r5\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:08:44.984664 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.984533    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:08:44.984664 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:44.984542    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb-isvc-pmml-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:08:45.737322 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.737295    2558 generic.go:356] "Generic (PLEG): container finished" podID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerID="f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd" exitCode=0
Jul 10 19:08:45.737705 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.737375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerDied","Data":"f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd"}
Jul 10 19:08:45.739444 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.739422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp" event={"ID":"2c45ce15-cd8f-4a6d-8962-f858cdfd81eb","Type":"ContainerDied","Data":"68b87159d1b5488b9ba0db76a03b5f2d6884c69d746c7ea34a3b6224a8a7d9df"}
Jul 10 19:08:45.739444 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.739436    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"
Jul 10 19:08:45.739620 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.739490    2558 scope.go:117] "RemoveContainer" containerID="886337e2db54a14b526f9a86a881d81c3716d7eeccc42dd9a3456174d5b1da76"
Jul 10 19:08:45.750296 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.750274    2558 scope.go:117] "RemoveContainer" containerID="1dfd2bea3461d493c735f15759f9d4d7fdc37510052b88b209ee211f74c2530b"
Jul 10 19:08:45.761045 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.761029    2558 scope.go:117] "RemoveContainer" containerID="4cba5ec85dc644e2f2cfae296f4d7be8dc4d1eed87c52a6b7bf578610fa74786"
Jul 10 19:08:45.768031 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.768011    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"]
Jul 10 19:08:45.774642 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:45.774619    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-56f7c95d45-b7mxp"]
Jul 10 19:08:46.744624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:46.744594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerStarted","Data":"2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073"}
Jul 10 19:08:46.745112 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:46.744655    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerStarted","Data":"bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9"}
Jul 10 19:08:46.745112 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:46.744890    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:46.764615 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:46.764574    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podStartSLOduration=2.10674518 podStartE2EDuration="5.764562297s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.657817117s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:46.762898747 +0000 UTC m=+1772.024563520" watchObservedRunningTime="2026-07-10 19:08:46.764562297 +0000 UTC m=+1772.026227069"
Jul 10 19:08:47.256960 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:47.256929    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" path="/var/lib/kubelet/pods/2c45ce15-cd8f-4a6d-8962-f858cdfd81eb/volumes"
Jul 10 19:08:47.749205 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:47.749176    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:47.750421 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:47.750394    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:08:48.752214 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:48.752179    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:08:53.757114 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:53.757087    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:08:53.757585 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:08:53.757557    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:09:03.757628 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:03.757586    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:09:13.758373 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:13.758337    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:09:15.279541 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:15.279513    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:09:15.281264 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:15.281245    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:09:23.757652 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:23.757617    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:09:33.757992 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:33.757950    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:09:43.758343 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:43.758298    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:09:53.757917 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:09:53.757880    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:10:03.256445 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:03.256416    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:10:12.660224 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.660190    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"]
Jul 10 19:10:12.662660 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.660442    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" containerID="cri-o://bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9" gracePeriod=30
Jul 10 19:10:12.662660 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.660474    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kube-rbac-proxy" containerID="cri-o://2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073" gracePeriod=30
Jul 10 19:10:12.677001 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.676978    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"]
Jul 10 19:10:12.677237 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677226    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container"
Jul 10 19:10:12.677280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677239    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container"
Jul 10 19:10:12.677280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677248    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kube-rbac-proxy"
Jul 10 19:10:12.677280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677253    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kube-rbac-proxy"
Jul 10 19:10:12.677280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677260    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="storage-initializer"
Jul 10 19:10:12.677280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677267    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="storage-initializer"
Jul 10 19:10:12.677434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677312    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kserve-container"
Jul 10 19:10:12.677434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.677319    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c45ce15-cd8f-4a6d-8962-f858cdfd81eb" containerName="kube-rbac-proxy"
Jul 10 19:10:12.680094 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.680078    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.682258 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.682238    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-846020-predictor-serving-cert\""
Jul 10 19:10:12.682329 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.682309    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-846020-kube-rbac-proxy-sar-config\""
Jul 10 19:10:12.691613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.691591    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"]
Jul 10 19:10:12.793965 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.793938    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-isvc-primary-846020-kube-rbac-proxy-sar-config\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.794067 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.793970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98t2t\" (UniqueName: \"kubernetes.io/projected/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kube-api-access-98t2t\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.794067 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.793997    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-proxy-tls\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.794067 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.794049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kserve-provision-location\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.894876 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.894845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-primary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-isvc-primary-846020-kube-rbac-proxy-sar-config\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.895010 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.894881    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98t2t\" (UniqueName: \"kubernetes.io/projected/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kube-api-access-98t2t\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.895010 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.894905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-proxy-tls\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.895132 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.895026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kserve-provision-location\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.895426 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.895405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kserve-provision-location\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.895577 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.895554    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-primary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-isvc-primary-846020-kube-rbac-proxy-sar-config\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.897298 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.897282    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-proxy-tls\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.906956 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.906938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98t2t\" (UniqueName: \"kubernetes.io/projected/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kube-api-access-98t2t\") pod \"isvc-primary-846020-predictor-5866d94d79-b2v76\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") " pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:12.990569 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:12.990511    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:13.057394 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:13.057368    2558 generic.go:356] "Generic (PLEG): container finished" podID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerID="2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073" exitCode=2
Jul 10 19:10:13.057543 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:13.057432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerDied","Data":"2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073"}
Jul 10 19:10:13.119688 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:13.119605    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"]
Jul 10 19:10:13.121777 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:10:13.121734    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6c16d9_fe28_4fb3_a5e6_785a6cd035dc.slice/crio-7d29ba516ab231b30833f9c7fde39c5fc99eeb3d5f9854edd9d9026c5737cc8b WatchSource:0}: Error finding container 7d29ba516ab231b30833f9c7fde39c5fc99eeb3d5f9854edd9d9026c5737cc8b: Status 404 returned error can't find the container with id 7d29ba516ab231b30833f9c7fde39c5fc99eeb3d5f9854edd9d9026c5737cc8b
Jul 10 19:10:13.253405 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:13.253361    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 10 19:10:13.753433 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:13.753393    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 10 19:10:14.061896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:14.061808    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerStarted","Data":"c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c"}
Jul 10 19:10:14.061896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:14.061866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerStarted","Data":"7d29ba516ab231b30833f9c7fde39c5fc99eeb3d5f9854edd9d9026c5737cc8b"}
Jul 10 19:10:16.000302 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.000275    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:10:16.070648 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.070554    2558 generic.go:356] "Generic (PLEG): container finished" podID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerID="bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9" exitCode=0
Jul 10 19:10:16.070648 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.070618    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"
Jul 10 19:10:16.070902 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.070645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerDied","Data":"bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9"}
Jul 10 19:10:16.070902 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.070770    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq" event={"ID":"d986c595-2a94-48fb-a851-1ffa7aab3389","Type":"ContainerDied","Data":"16238047afb99c9886de1fd95a10805b1cfbfa7cdaac708820657e5011226099"}
Jul 10 19:10:16.070902 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.070812    2558 scope.go:117] "RemoveContainer" containerID="2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073"
Jul 10 19:10:16.078442 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.078414    2558 scope.go:117] "RemoveContainer" containerID="bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9"
Jul 10 19:10:16.085326 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.085311    2558 scope.go:117] "RemoveContainer" containerID="f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd"
Jul 10 19:10:16.091940 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.091924    2558 scope.go:117] "RemoveContainer" containerID="2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073"
Jul 10 19:10:16.092165 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:10:16.092148    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073\": container with ID starting with 2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073 not found: ID does not exist" containerID="2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073"
Jul 10 19:10:16.092221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.092172    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073"} err="failed to get container status \"2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073\": rpc error: code = NotFound desc = could not find container \"2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073\": container with ID starting with 2bd41a4eb3aea2ee9722b43f478bf1b24361420f5765c8e2f0c1df0b0072f073 not found: ID does not exist"
Jul 10 19:10:16.092221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.092189    2558 scope.go:117] "RemoveContainer" containerID="bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9"
Jul 10 19:10:16.092399 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:10:16.092383    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9\": container with ID starting with bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9 not found: ID does not exist" containerID="bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9"
Jul 10 19:10:16.092439 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.092403    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9"} err="failed to get container status \"bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9\": rpc error: code = NotFound desc = could not find container \"bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9\": container with ID starting with bee045ad784108b0f45535e53fa2d3eb1cb478da7fb940be218d3dfb9dd036d9 not found: ID does not exist"
Jul 10 19:10:16.092439 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.092418    2558 scope.go:117] "RemoveContainer" containerID="f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd"
Jul 10 19:10:16.092615 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:10:16.092601    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd\": container with ID starting with f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd not found: ID does not exist" containerID="f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd"
Jul 10 19:10:16.092655 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.092620    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd"} err="failed to get container status \"f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd\": rpc error: code = NotFound desc = could not find container \"f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd\": container with ID starting with f79f87fa5f93548ee52725792c2ae2b709623903d9a3816516df06097ad88ebd not found: ID does not exist"
Jul 10 19:10:16.121918 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.121897    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d986c595-2a94-48fb-a851-1ffa7aab3389-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") pod \"d986c595-2a94-48fb-a851-1ffa7aab3389\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") "
Jul 10 19:10:16.121990 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.121931    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d986c595-2a94-48fb-a851-1ffa7aab3389-proxy-tls\") pod \"d986c595-2a94-48fb-a851-1ffa7aab3389\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") "
Jul 10 19:10:16.122030 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.121988    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l84bn\" (UniqueName: \"kubernetes.io/projected/d986c595-2a94-48fb-a851-1ffa7aab3389-kube-api-access-l84bn\") pod \"d986c595-2a94-48fb-a851-1ffa7aab3389\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") "
Jul 10 19:10:16.122030 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.122013    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d986c595-2a94-48fb-a851-1ffa7aab3389-kserve-provision-location\") pod \"d986c595-2a94-48fb-a851-1ffa7aab3389\" (UID: \"d986c595-2a94-48fb-a851-1ffa7aab3389\") "
Jul 10 19:10:16.122367 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.122346    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d986c595-2a94-48fb-a851-1ffa7aab3389-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config") pod "d986c595-2a94-48fb-a851-1ffa7aab3389" (UID: "d986c595-2a94-48fb-a851-1ffa7aab3389"). InnerVolumeSpecName "isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:10:16.122428 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.122346    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d986c595-2a94-48fb-a851-1ffa7aab3389-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d986c595-2a94-48fb-a851-1ffa7aab3389" (UID: "d986c595-2a94-48fb-a851-1ffa7aab3389"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:10:16.123982 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.123963    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d986c595-2a94-48fb-a851-1ffa7aab3389-kube-api-access-l84bn" (OuterVolumeSpecName: "kube-api-access-l84bn") pod "d986c595-2a94-48fb-a851-1ffa7aab3389" (UID: "d986c595-2a94-48fb-a851-1ffa7aab3389"). InnerVolumeSpecName "kube-api-access-l84bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:10:16.124044 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.124009    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d986c595-2a94-48fb-a851-1ffa7aab3389-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d986c595-2a94-48fb-a851-1ffa7aab3389" (UID: "d986c595-2a94-48fb-a851-1ffa7aab3389"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:10:16.223267 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.223234    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d986c595-2a94-48fb-a851-1ffa7aab3389-isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:10:16.223267 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.223263    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d986c595-2a94-48fb-a851-1ffa7aab3389-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:10:16.223267 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.223274    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84bn\" (UniqueName: \"kubernetes.io/projected/d986c595-2a94-48fb-a851-1ffa7aab3389-kube-api-access-l84bn\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:10:16.223484 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.223283    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d986c595-2a94-48fb-a851-1ffa7aab3389-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:10:16.392012 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.391982    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"]
Jul 10 19:10:16.397433 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:16.397413    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-7859dbdb75-nrtfq"]
Jul 10 19:10:17.075483 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:17.075455    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerID="c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c" exitCode=0
Jul 10 19:10:17.075872 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:17.075525    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerDied","Data":"c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c"}
Jul 10 19:10:17.257532 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:17.257503    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" path="/var/lib/kubelet/pods/d986c595-2a94-48fb-a851-1ffa7aab3389/volumes"
Jul 10 19:10:18.083144 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:18.083107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerStarted","Data":"29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd"}
Jul 10 19:10:18.083525 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:18.083168    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerStarted","Data":"1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a"}
Jul 10 19:10:18.083525 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:18.083364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:18.105624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:18.105565    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podStartSLOduration=2.34794186 podStartE2EDuration="6.105544491s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.757602631s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:18.10332542 +0000 UTC m=+1863.364990230" watchObservedRunningTime="2026-07-10 19:10:18.105544491 +0000 UTC m=+1863.367209267"
Jul 10 19:10:19.086491 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:19.086456    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:19.087515 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:19.087488    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:10:20.090177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:20.090142    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:10:25.095145 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:25.095113    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:10:25.095778 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:25.095725    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:10:35.095890 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:35.095854    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:10:45.096386 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:45.096298    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:10:55.096467 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:10:55.096423    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:11:05.095947 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:05.095912    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:11:15.095751 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:15.095701    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 10 19:11:25.096592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:25.096564    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.810403    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"]
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811031    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="storage-initializer"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811057    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="storage-initializer"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811073    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811082    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811112    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kube-rbac-proxy"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811121    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kube-rbac-proxy"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811231    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kserve-container"
Jul 10 19:11:32.812580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.811244    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d986c595-2a94-48fb-a851-1ffa7aab3389" containerName="kube-rbac-proxy"
Jul 10 19:11:32.815542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.815519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.817841 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.817815    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-846020-predictor-serving-cert\""
Jul 10 19:11:32.817965 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.817842    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-846020-kube-rbac-proxy-sar-config\""
Jul 10 19:11:32.817965 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.817880    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:11:32.817965 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.817880    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-846020\""
Jul 10 19:11:32.818181 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.818165    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-846020-dockercfg-fsn89\""
Jul 10 19:11:32.822414 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.822389    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"]
Jul 10 19:11:32.845019 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.844995    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-cabundle-cert\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.845132 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.845030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8fj\" (UniqueName: \"kubernetes.io/projected/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kube-api-access-hr8fj\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.845132 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.845057    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-isvc-secondary-846020-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.845132 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.845082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-proxy-tls\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.845282 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.845165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kserve-provision-location\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.946383 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.946354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-cabundle-cert\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.946542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.946398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8fj\" (UniqueName: \"kubernetes.io/projected/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kube-api-access-hr8fj\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.946542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.946439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-secondary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-isvc-secondary-846020-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.946542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.946477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-proxy-tls\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.946542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.946529    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kserve-provision-location\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.946904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.946884    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kserve-provision-location\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.947053 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.947024    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-cabundle-cert\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.947165 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.947116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-secondary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-isvc-secondary-846020-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.948927 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.948910    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-proxy-tls\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:32.954973 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:32.954951    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8fj\" (UniqueName: \"kubernetes.io/projected/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kube-api-access-hr8fj\") pod \"isvc-secondary-846020-predictor-6d467dd688-6c578\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") " pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:33.127772 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:33.127647    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:33.255475 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:11:33.255445    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d6886a_9dcd_4ac1_a4e5_153ec83f3adf.slice/crio-bd35caf7ff8f200609e1bfd0a746592bdd696c77f7d37284d28b2e0badcffb60 WatchSource:0}: Error finding container bd35caf7ff8f200609e1bfd0a746592bdd696c77f7d37284d28b2e0badcffb60: Status 404 returned error can't find the container with id bd35caf7ff8f200609e1bfd0a746592bdd696c77f7d37284d28b2e0badcffb60
Jul 10 19:11:33.257055 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:33.257033    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"]
Jul 10 19:11:33.345848 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:33.345819    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578" event={"ID":"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf","Type":"ContainerStarted","Data":"01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9"}
Jul 10 19:11:33.346010 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:33.345895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578" event={"ID":"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf","Type":"ContainerStarted","Data":"bd35caf7ff8f200609e1bfd0a746592bdd696c77f7d37284d28b2e0badcffb60"}
Jul 10 19:11:37.360584 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:37.360558    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-846020-predictor-6d467dd688-6c578_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/storage-initializer/0.log"
Jul 10 19:11:37.361034 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:37.360598    2558 generic.go:356] "Generic (PLEG): container finished" podID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerID="01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9" exitCode=1
Jul 10 19:11:37.361034 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:37.360673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578" event={"ID":"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf","Type":"ContainerDied","Data":"01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9"}
Jul 10 19:11:38.365801 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:38.365772    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-846020-predictor-6d467dd688-6c578_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/storage-initializer/0.log"
Jul 10 19:11:38.366291 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:38.365871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578" event={"ID":"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf","Type":"ContainerStarted","Data":"589c71cc2687438b8fd132401e495e0362c485b8798b7d368e99bdca93a3328f"}
Jul 10 19:11:39.371009 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:39.370985    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-846020-predictor-6d467dd688-6c578_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/storage-initializer/1.log"
Jul 10 19:11:39.371371 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:39.371347    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-846020-predictor-6d467dd688-6c578_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/storage-initializer/0.log"
Jul 10 19:11:39.371423 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:39.371378    2558 generic.go:356] "Generic (PLEG): container finished" podID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerID="589c71cc2687438b8fd132401e495e0362c485b8798b7d368e99bdca93a3328f" exitCode=1
Jul 10 19:11:39.371469 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:39.371431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578" event={"ID":"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf","Type":"ContainerDied","Data":"589c71cc2687438b8fd132401e495e0362c485b8798b7d368e99bdca93a3328f"}
Jul 10 19:11:39.371520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:39.371482    2558 scope.go:117] "RemoveContainer" containerID="01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9"
Jul 10 19:11:39.371865 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:39.371836    2558 scope.go:117] "RemoveContainer" containerID="01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9"
Jul 10 19:11:39.381835 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:11:39.381815    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-846020-predictor-6d467dd688-6c578_kserve-ci-e2e-test(71d6886a-9dcd-4ac1-a4e5-153ec83f3adf)\"" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf"
Jul 10 19:11:39.382002 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:11:39.381979    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-846020-predictor-6d467dd688-6c578_kserve-ci-e2e-test_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf_0 in pod sandbox bd35caf7ff8f200609e1bfd0a746592bdd696c77f7d37284d28b2e0badcffb60 from index: no such id: '01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9'" containerID="01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9"
Jul 10 19:11:39.382070 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:39.382015    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-846020-predictor-6d467dd688-6c578_kserve-ci-e2e-test_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf_0 in pod sandbox bd35caf7ff8f200609e1bfd0a746592bdd696c77f7d37284d28b2e0badcffb60 from index: no such id: '01290feb42c313d20622aed6ec27e885591932c73f85e9fd647d3def4a1f01d9'"
Jul 10 19:11:40.377033 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:40.377005    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-846020-predictor-6d467dd688-6c578_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/storage-initializer/1.log"
Jul 10 19:11:48.894917 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:48.894886    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"]
Jul 10 19:11:48.977305 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:48.977275    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"]
Jul 10 19:11:48.977547 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:48.977521    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container" containerID="cri-o://1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a" gracePeriod=30
Jul 10 19:11:48.977624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:48.977549    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kube-rbac-proxy" containerID="cri-o://29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd" gracePeriod=30
Jul 10 19:11:49.011850 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.011823    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"]
Jul 10 19:11:49.021037 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.021015    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.023319 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.023296    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6d20ea-predictor-serving-cert\""
Jul 10 19:11:49.023438 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.023341    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-6d20ea\""
Jul 10 19:11:49.023438 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.023298    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\""
Jul 10 19:11:49.023438 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.023367    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-6d20ea-dockercfg-rljjv\""
Jul 10 19:11:49.027449 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.027415    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"]
Jul 10 19:11:49.066833 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.066800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88d57251-4d12-46e6-9897-16ba174679f5-kserve-provision-location\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.067007 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.066850    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-cabundle-cert\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.067007 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.066931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88d57251-4d12-46e6-9897-16ba174679f5-proxy-tls\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.067101 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.067012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.067101 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.067047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ntf\" (UniqueName: \"kubernetes.io/projected/88d57251-4d12-46e6-9897-16ba174679f5-kube-api-access-p2ntf\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.095503 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.095485    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-846020-predictor-6d467dd688-6c578_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/storage-initializer/1.log"
Jul 10 19:11:49.097609 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.097592    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:49.167981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.167909    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-cabundle-cert\") pod \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") "
Jul 10 19:11:49.167981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.167953    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-isvc-secondary-846020-kube-rbac-proxy-sar-config\") pod \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") "
Jul 10 19:11:49.168190 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.167981    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kserve-provision-location\") pod \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") "
Jul 10 19:11:49.168190 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168049    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr8fj\" (UniqueName: \"kubernetes.io/projected/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kube-api-access-hr8fj\") pod \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") "
Jul 10 19:11:49.168190 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168080    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-proxy-tls\") pod \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\" (UID: \"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf\") "
Jul 10 19:11:49.168190 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.168399 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ntf\" (UniqueName: \"kubernetes.io/projected/88d57251-4d12-46e6-9897-16ba174679f5-kube-api-access-p2ntf\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.168399 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88d57251-4d12-46e6-9897-16ba174679f5-kserve-provision-location\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.168399 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168270    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" (UID: "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:11:49.168399 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-cabundle-cert\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.168593 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88d57251-4d12-46e6-9897-16ba174679f5-proxy-tls\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.168593 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168448    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-isvc-secondary-846020-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-846020-kube-rbac-proxy-sar-config") pod "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" (UID: "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf"). InnerVolumeSpecName "isvc-secondary-846020-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:49.168593 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168469    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:49.168781 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.168678    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" (UID: "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:49.169212 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.169162    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88d57251-4d12-46e6-9897-16ba174679f5-kserve-provision-location\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.169425 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.169401    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-cabundle-cert\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.169613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.169592    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.170466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.170442    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" (UID: "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:11:49.170702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.170684    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kube-api-access-hr8fj" (OuterVolumeSpecName: "kube-api-access-hr8fj") pod "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" (UID: "71d6886a-9dcd-4ac1-a4e5-153ec83f3adf"). InnerVolumeSpecName "kube-api-access-hr8fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:11:49.171209 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.171186    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88d57251-4d12-46e6-9897-16ba174679f5-proxy-tls\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.175912 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.175891    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ntf\" (UniqueName: \"kubernetes.io/projected/88d57251-4d12-46e6-9897-16ba174679f5-kube-api-access-p2ntf\") pod \"isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") " pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.269025 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.269003    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-cabundle-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:49.269025 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.269025    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-isvc-secondary-846020-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:49.269155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.269035    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr8fj\" (UniqueName: \"kubernetes.io/projected/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-kube-api-access-hr8fj\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:49.269155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.269045    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:49.332202 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.332173    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:11:49.410773 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.410669    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerID="29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd" exitCode=2
Jul 10 19:11:49.410773 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.410728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerDied","Data":"29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd"}
Jul 10 19:11:49.411898 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.411875    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-846020-predictor-6d467dd688-6c578_71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/storage-initializer/1.log"
Jul 10 19:11:49.412031 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.411928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578" event={"ID":"71d6886a-9dcd-4ac1-a4e5-153ec83f3adf","Type":"ContainerDied","Data":"bd35caf7ff8f200609e1bfd0a746592bdd696c77f7d37284d28b2e0badcffb60"}
Jul 10 19:11:49.412031 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.411988    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"
Jul 10 19:11:49.412141 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.411991    2558 scope.go:117] "RemoveContainer" containerID="589c71cc2687438b8fd132401e495e0362c485b8798b7d368e99bdca93a3328f"
Jul 10 19:11:49.445208 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.445161    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"]
Jul 10 19:11:49.451199 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.451083    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-846020-predictor-6d467dd688-6c578"]
Jul 10 19:11:49.463728 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:49.463701    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"]
Jul 10 19:11:49.466706 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:11:49.466681    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d57251_4d12_46e6_9897_16ba174679f5.slice/crio-e4a00c097e318da57267234d62853527f64d714ba55f2305e8be6cec1b5f20df WatchSource:0}: Error finding container e4a00c097e318da57267234d62853527f64d714ba55f2305e8be6cec1b5f20df: Status 404 returned error can't find the container with id e4a00c097e318da57267234d62853527f64d714ba55f2305e8be6cec1b5f20df
Jul 10 19:11:50.091040 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:50.090992    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 10 19:11:50.419008 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:50.418924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8" event={"ID":"88d57251-4d12-46e6-9897-16ba174679f5","Type":"ContainerStarted","Data":"eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9"}
Jul 10 19:11:50.419008 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:50.418986    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8" event={"ID":"88d57251-4d12-46e6-9897-16ba174679f5","Type":"ContainerStarted","Data":"e4a00c097e318da57267234d62853527f64d714ba55f2305e8be6cec1b5f20df"}
Jul 10 19:11:51.256538 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:51.256504    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" path="/var/lib/kubelet/pods/71d6886a-9dcd-4ac1-a4e5-153ec83f3adf/volumes"
Jul 10 19:11:53.129970 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.129949    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:11:53.199326 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.199254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kserve-provision-location\") pod \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") "
Jul 10 19:11:53.199326 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.199320    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-proxy-tls\") pod \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") "
Jul 10 19:11:53.199496 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.199387    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-primary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-isvc-primary-846020-kube-rbac-proxy-sar-config\") pod \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") "
Jul 10 19:11:53.199496 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.199418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98t2t\" (UniqueName: \"kubernetes.io/projected/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kube-api-access-98t2t\") pod \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\" (UID: \"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc\") "
Jul 10 19:11:53.199609 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.199585    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" (UID: "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:11:53.199764 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.199718    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-isvc-primary-846020-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-846020-kube-rbac-proxy-sar-config") pod "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" (UID: "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc"). InnerVolumeSpecName "isvc-primary-846020-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:11:53.201432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.201411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" (UID: "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:11:53.201501 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.201459    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kube-api-access-98t2t" (OuterVolumeSpecName: "kube-api-access-98t2t") pod "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" (UID: "fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc"). InnerVolumeSpecName "kube-api-access-98t2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:11:53.299849 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.299826    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:53.299849 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.299848    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-846020-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-isvc-primary-846020-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:53.299989 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.299858    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98t2t\" (UniqueName: \"kubernetes.io/projected/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kube-api-access-98t2t\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:53.299989 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.299868    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:11:53.431892 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.431854    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerID="1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a" exitCode=0
Jul 10 19:11:53.432069 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.431916    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"
Jul 10 19:11:53.432069 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.431936    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerDied","Data":"1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a"}
Jul 10 19:11:53.432069 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.431994    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76" event={"ID":"fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc","Type":"ContainerDied","Data":"7d29ba516ab231b30833f9c7fde39c5fc99eeb3d5f9854edd9d9026c5737cc8b"}
Jul 10 19:11:53.432069 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.432023    2558 scope.go:117] "RemoveContainer" containerID="29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd"
Jul 10 19:11:53.439963 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.439946    2558 scope.go:117] "RemoveContainer" containerID="1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a"
Jul 10 19:11:53.447420 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.447402    2558 scope.go:117] "RemoveContainer" containerID="c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c"
Jul 10 19:11:53.448529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.448508    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"]
Jul 10 19:11:53.451880 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.451823    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-846020-predictor-5866d94d79-b2v76"]
Jul 10 19:11:53.454674 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.454657    2558 scope.go:117] "RemoveContainer" containerID="29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd"
Jul 10 19:11:53.454953 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:11:53.454935    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd\": container with ID starting with 29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd not found: ID does not exist" containerID="29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd"
Jul 10 19:11:53.455023 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.454961    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd"} err="failed to get container status \"29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd\": rpc error: code = NotFound desc = could not find container \"29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd\": container with ID starting with 29e5ee1ddc183a565de643fd997663e6b56898da80defb5ef109ccf39c890ddd not found: ID does not exist"
Jul 10 19:11:53.455023 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.454982    2558 scope.go:117] "RemoveContainer" containerID="1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a"
Jul 10 19:11:53.455216 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:11:53.455201    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a\": container with ID starting with 1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a not found: ID does not exist" containerID="1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a"
Jul 10 19:11:53.455267 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.455220    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a"} err="failed to get container status \"1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a\": rpc error: code = NotFound desc = could not find container \"1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a\": container with ID starting with 1cf5350ecda01505746d55e49d54b91944ac62f5597c4443d2412370afe9383a not found: ID does not exist"
Jul 10 19:11:53.455267 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.455235    2558 scope.go:117] "RemoveContainer" containerID="c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c"
Jul 10 19:11:53.455454 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:11:53.455440    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c\": container with ID starting with c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c not found: ID does not exist" containerID="c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c"
Jul 10 19:11:53.455497 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:53.455457    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c"} err="failed to get container status \"c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c\": rpc error: code = NotFound desc = could not find container \"c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c\": container with ID starting with c336964ed5fb1a6e3018cac26189f9052412c5f88b90025d1f12edb448df7a8c not found: ID does not exist"
Jul 10 19:11:55.256482 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:55.256451    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" path="/var/lib/kubelet/pods/fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc/volumes"
Jul 10 19:11:56.443925 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:56.443843    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8_88d57251-4d12-46e6-9897-16ba174679f5/storage-initializer/0.log"
Jul 10 19:11:56.443925 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:56.443883    2558 generic.go:356] "Generic (PLEG): container finished" podID="88d57251-4d12-46e6-9897-16ba174679f5" containerID="eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9" exitCode=1
Jul 10 19:11:56.444425 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:56.443934    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8" event={"ID":"88d57251-4d12-46e6-9897-16ba174679f5","Type":"ContainerDied","Data":"eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9"}
Jul 10 19:11:57.448715 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:57.448685    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8_88d57251-4d12-46e6-9897-16ba174679f5/storage-initializer/0.log"
Jul 10 19:11:57.449075 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:57.448800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8" event={"ID":"88d57251-4d12-46e6-9897-16ba174679f5","Type":"ContainerStarted","Data":"fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c"}
Jul 10 19:11:58.991761 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:58.991691    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"]
Jul 10 19:11:58.992198 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:58.991993    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8" podUID="88d57251-4d12-46e6-9897-16ba174679f5" containerName="storage-initializer" containerID="cri-o://fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c" gracePeriod=30
Jul 10 19:11:59.344618 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344585    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"]
Jul 10 19:11:59.344867 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344854    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerName="storage-initializer"
Jul 10 19:11:59.344914 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344869    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerName="storage-initializer"
Jul 10 19:11:59.344914 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344890    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="storage-initializer"
Jul 10 19:11:59.344914 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344895    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="storage-initializer"
Jul 10 19:11:59.344914 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344903    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container"
Jul 10 19:11:59.344914 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344908    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container"
Jul 10 19:11:59.345064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344918    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kube-rbac-proxy"
Jul 10 19:11:59.345064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344924    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kube-rbac-proxy"
Jul 10 19:11:59.345064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344962    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerName="storage-initializer"
Jul 10 19:11:59.345064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344971    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kserve-container"
Jul 10 19:11:59.345064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.344977    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa6c16d9-fe28-4fb3-a5e6-785a6cd035dc" containerName="kube-rbac-proxy"
Jul 10 19:11:59.345064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.345030    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerName="storage-initializer"
Jul 10 19:11:59.345064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.345036    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerName="storage-initializer"
Jul 10 19:11:59.345284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.345076    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="71d6886a-9dcd-4ac1-a4e5-153ec83f3adf" containerName="storage-initializer"
Jul 10 19:11:59.347988 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.347973    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.350386 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.350365    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mtzvk\""
Jul 10 19:11:59.350505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.350364    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 10 19:11:59.350547 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.350504    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 10 19:11:59.358503 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.358480    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"]
Jul 10 19:11:59.444171 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.444140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a4028a-2e82-4f38-95db-84f72a6cd286-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.444337 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.444177    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a4028a-2e82-4f38-95db-84f72a6cd286-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.444337 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.444202    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmtm\" (UniqueName: \"kubernetes.io/projected/17a4028a-2e82-4f38-95db-84f72a6cd286-kube-api-access-lnmtm\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.444459 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.444351    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a4028a-2e82-4f38-95db-84f72a6cd286-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.544956 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.544913    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmtm\" (UniqueName: \"kubernetes.io/projected/17a4028a-2e82-4f38-95db-84f72a6cd286-kube-api-access-lnmtm\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.545137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.544991    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a4028a-2e82-4f38-95db-84f72a6cd286-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.545137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.545022    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a4028a-2e82-4f38-95db-84f72a6cd286-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.545137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.545053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a4028a-2e82-4f38-95db-84f72a6cd286-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.545403 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.545383    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a4028a-2e82-4f38-95db-84f72a6cd286-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.545699 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.545680    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a4028a-2e82-4f38-95db-84f72a6cd286-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.547410 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.547393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a4028a-2e82-4f38-95db-84f72a6cd286-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.553796 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.553778    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmtm\" (UniqueName: \"kubernetes.io/projected/17a4028a-2e82-4f38-95db-84f72a6cd286-kube-api-access-lnmtm\") pod \"isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.658830 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.658754    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:11:59.784540 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:11:59.784499    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"]
Jul 10 19:11:59.788010 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:11:59.787968    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a4028a_2e82_4f38_95db_84f72a6cd286.slice/crio-f50fe8c1267527fe0f1911d492238220bb79e43ccc430a65b588b648d7f2515a WatchSource:0}: Error finding container f50fe8c1267527fe0f1911d492238220bb79e43ccc430a65b588b648d7f2515a: Status 404 returned error can't find the container with id f50fe8c1267527fe0f1911d492238220bb79e43ccc430a65b588b648d7f2515a
Jul 10 19:12:00.224299 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.224279    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8_88d57251-4d12-46e6-9897-16ba174679f5/storage-initializer/1.log"
Jul 10 19:12:00.224696 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.224678    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8_88d57251-4d12-46e6-9897-16ba174679f5/storage-initializer/0.log"
Jul 10 19:12:00.226990 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.226972    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:12:00.352141 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352057    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-cabundle-cert\") pod \"88d57251-4d12-46e6-9897-16ba174679f5\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") "
Jul 10 19:12:00.352141 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352127    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88d57251-4d12-46e6-9897-16ba174679f5-kserve-provision-location\") pod \"88d57251-4d12-46e6-9897-16ba174679f5\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") "
Jul 10 19:12:00.352355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352153    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2ntf\" (UniqueName: \"kubernetes.io/projected/88d57251-4d12-46e6-9897-16ba174679f5-kube-api-access-p2ntf\") pod \"88d57251-4d12-46e6-9897-16ba174679f5\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") "
Jul 10 19:12:00.352355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352180    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88d57251-4d12-46e6-9897-16ba174679f5-proxy-tls\") pod \"88d57251-4d12-46e6-9897-16ba174679f5\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") "
Jul 10 19:12:00.352355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352214    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\") pod \"88d57251-4d12-46e6-9897-16ba174679f5\" (UID: \"88d57251-4d12-46e6-9897-16ba174679f5\") "
Jul 10 19:12:00.352524 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352449    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d57251-4d12-46e6-9897-16ba174679f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88d57251-4d12-46e6-9897-16ba174679f5" (UID: "88d57251-4d12-46e6-9897-16ba174679f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:12:00.352620 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352579    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88d57251-4d12-46e6-9897-16ba174679f5" (UID: "88d57251-4d12-46e6-9897-16ba174679f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:12:00.352717 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.352630    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config") pod "88d57251-4d12-46e6-9897-16ba174679f5" (UID: "88d57251-4d12-46e6-9897-16ba174679f5"). InnerVolumeSpecName "isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:12:00.354367 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.354343    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d57251-4d12-46e6-9897-16ba174679f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88d57251-4d12-46e6-9897-16ba174679f5" (UID: "88d57251-4d12-46e6-9897-16ba174679f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:12:00.354453 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.354391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d57251-4d12-46e6-9897-16ba174679f5-kube-api-access-p2ntf" (OuterVolumeSpecName: "kube-api-access-p2ntf") pod "88d57251-4d12-46e6-9897-16ba174679f5" (UID: "88d57251-4d12-46e6-9897-16ba174679f5"). InnerVolumeSpecName "kube-api-access-p2ntf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:12:00.452887 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.452838    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-cabundle-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:12:00.452887 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.452878    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88d57251-4d12-46e6-9897-16ba174679f5-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:12:00.452887 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.452892    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2ntf\" (UniqueName: \"kubernetes.io/projected/88d57251-4d12-46e6-9897-16ba174679f5-kube-api-access-p2ntf\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:12:00.453128 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.452905    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88d57251-4d12-46e6-9897-16ba174679f5-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:12:00.453128 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.452918    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88d57251-4d12-46e6-9897-16ba174679f5-isvc-init-fail-6d20ea-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:12:00.462925 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.462900    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8_88d57251-4d12-46e6-9897-16ba174679f5/storage-initializer/1.log"
Jul 10 19:12:00.463350 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.463335    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8_88d57251-4d12-46e6-9897-16ba174679f5/storage-initializer/0.log"
Jul 10 19:12:00.463417 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.463367    2558 generic.go:356] "Generic (PLEG): container finished" podID="88d57251-4d12-46e6-9897-16ba174679f5" containerID="fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c" exitCode=1
Jul 10 19:12:00.463465 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.463426    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"
Jul 10 19:12:00.463465 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.463432    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8" event={"ID":"88d57251-4d12-46e6-9897-16ba174679f5","Type":"ContainerDied","Data":"fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c"}
Jul 10 19:12:00.463540 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.463492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8" event={"ID":"88d57251-4d12-46e6-9897-16ba174679f5","Type":"ContainerDied","Data":"e4a00c097e318da57267234d62853527f64d714ba55f2305e8be6cec1b5f20df"}
Jul 10 19:12:00.463540 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.463532    2558 scope.go:117] "RemoveContainer" containerID="fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c"
Jul 10 19:12:00.465037 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.465014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerStarted","Data":"2c04c020b539973e0d4e06be115dd44d184579152af5599b9a8496790a563fe5"}
Jul 10 19:12:00.465143 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.465068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerStarted","Data":"f50fe8c1267527fe0f1911d492238220bb79e43ccc430a65b588b648d7f2515a"}
Jul 10 19:12:00.471849 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.471834    2558 scope.go:117] "RemoveContainer" containerID="eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9"
Jul 10 19:12:00.478582 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.478567    2558 scope.go:117] "RemoveContainer" containerID="fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c"
Jul 10 19:12:00.478842 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:12:00.478825    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c\": container with ID starting with fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c not found: ID does not exist" containerID="fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c"
Jul 10 19:12:00.478921 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.478852    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c"} err="failed to get container status \"fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c\": rpc error: code = NotFound desc = could not find container \"fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c\": container with ID starting with fca34e810d723323dc986ae3ddcfca8cb07463ac6a1c51487b422555b4d2ce4c not found: ID does not exist"
Jul 10 19:12:00.478921 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.478874    2558 scope.go:117] "RemoveContainer" containerID="eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9"
Jul 10 19:12:00.479138 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:12:00.479120    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9\": container with ID starting with eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9 not found: ID does not exist" containerID="eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9"
Jul 10 19:12:00.479177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.479145    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9"} err="failed to get container status \"eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9\": rpc error: code = NotFound desc = could not find container \"eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9\": container with ID starting with eaaae52fae536efc32d5d4dbc0e60b172895e73e61bbb3759428130eb4ce57c9 not found: ID does not exist"
Jul 10 19:12:00.513137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.513115    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"]
Jul 10 19:12:00.519264 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:00.519244    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-6d20ea-predictor-5b45b978b4-jpth8"]
Jul 10 19:12:01.257055 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:01.257027    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d57251-4d12-46e6-9897-16ba174679f5" path="/var/lib/kubelet/pods/88d57251-4d12-46e6-9897-16ba174679f5/volumes"
Jul 10 19:12:03.478839 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:03.478802    2558 generic.go:356] "Generic (PLEG): container finished" podID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerID="2c04c020b539973e0d4e06be115dd44d184579152af5599b9a8496790a563fe5" exitCode=0
Jul 10 19:12:03.479172 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:03.478873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerDied","Data":"2c04c020b539973e0d4e06be115dd44d184579152af5599b9a8496790a563fe5"}
Jul 10 19:12:03.480047 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:03.480033    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:12:26.583680 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:26.583648    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerStarted","Data":"3fc18eb372dacda3e63ee22031b210b34d22aa93d60d724aae14a6fe34bcf0e2"}
Jul 10 19:12:26.584106 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:26.583702    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerStarted","Data":"5ef2fead58c11a4e041acc7280ecde26500eac0d136b34639bd50053f759619c"}
Jul 10 19:12:26.584106 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:26.583983    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:12:26.584191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:26.584167    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:12:26.584938 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:26.584916    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:12:26.607166 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:26.607119    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podStartSLOduration=1.849971455 podStartE2EDuration="27.607108069s" totalImagesPullingTime="22.18530037s" totalInitContainerRuntime="3.571836244s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:12:26.605643747 +0000 UTC m=+1991.867308526" watchObservedRunningTime="2026-07-10 19:12:26.607108069 +0000 UTC m=+1991.868772843"
Jul 10 19:12:27.588353 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:27.588312    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:12:28.592470 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:28.592432    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:12:33.596355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:33.596329    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:12:33.596821 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:33.596794    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:12:43.597293 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:43.597255    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:12:53.596979 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:12:53.596935    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:13:03.597224 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:03.597185    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:13:13.597001 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:13.596961    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:13:23.596768 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:23.596706    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:13:33.597133 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:33.597089    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:13:43.256670 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:43.256598    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:13:50.349825 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.349790    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"]
Jul 10 19:13:50.350284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.350204    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d57251-4d12-46e6-9897-16ba174679f5" containerName="storage-initializer"
Jul 10 19:13:50.350284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.350221    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d57251-4d12-46e6-9897-16ba174679f5" containerName="storage-initializer"
Jul 10 19:13:50.350284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.350235    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d57251-4d12-46e6-9897-16ba174679f5" containerName="storage-initializer"
Jul 10 19:13:50.350284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.350243    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d57251-4d12-46e6-9897-16ba174679f5" containerName="storage-initializer"
Jul 10 19:13:50.350509 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.350325    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d57251-4d12-46e6-9897-16ba174679f5" containerName="storage-initializer"
Jul 10 19:13:50.350509 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.350474    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d57251-4d12-46e6-9897-16ba174679f5" containerName="storage-initializer"
Jul 10 19:13:50.352281 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.352261    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.355546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.355525    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 10 19:13:50.355977 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.355961    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 10 19:13:50.359591 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.359569    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"]
Jul 10 19:13:50.360025 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.359999    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" containerID="cri-o://5ef2fead58c11a4e041acc7280ecde26500eac0d136b34639bd50053f759619c" gracePeriod=30
Jul 10 19:13:50.360195 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.360165    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kube-rbac-proxy" containerID="cri-o://3fc18eb372dacda3e63ee22031b210b34d22aa93d60d724aae14a6fe34bcf0e2" gracePeriod=30
Jul 10 19:13:50.369103 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.369083    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"]
Jul 10 19:13:50.472768 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.472720    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4n9d\" (UniqueName: \"kubernetes.io/projected/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kube-api-access-q4n9d\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.472943 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.472830    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3ff6f07-2f9d-4985-ab5d-ee641349b567-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.472943 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.472861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3ff6f07-2f9d-4985-ab5d-ee641349b567-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.472943 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.472908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.573478 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.573447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.573619 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.573506    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4n9d\" (UniqueName: \"kubernetes.io/projected/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kube-api-access-q4n9d\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.573619 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.573536    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3ff6f07-2f9d-4985-ab5d-ee641349b567-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.573619 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.573556    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3ff6f07-2f9d-4985-ab5d-ee641349b567-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.573851 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.573823    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.574177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.574157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3ff6f07-2f9d-4985-ab5d-ee641349b567-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.575919 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.575898    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3ff6f07-2f9d-4985-ab5d-ee641349b567-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.582272 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.582250    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4n9d\" (UniqueName: \"kubernetes.io/projected/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kube-api-access-q4n9d\") pod \"isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.665024 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.664948    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:50.791736 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.791706    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"]
Jul 10 19:13:50.795113 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:13:50.795090    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ff6f07_2f9d_4985_ab5d_ee641349b567.slice/crio-b1a40ff3e4accdfd98cb32b25864d8efbb0b12cedfe6bbaadd0588d869806eaa WatchSource:0}: Error finding container b1a40ff3e4accdfd98cb32b25864d8efbb0b12cedfe6bbaadd0588d869806eaa: Status 404 returned error can't find the container with id b1a40ff3e4accdfd98cb32b25864d8efbb0b12cedfe6bbaadd0588d869806eaa
Jul 10 19:13:50.886282 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.886247    2558 generic.go:356] "Generic (PLEG): container finished" podID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerID="3fc18eb372dacda3e63ee22031b210b34d22aa93d60d724aae14a6fe34bcf0e2" exitCode=2
Jul 10 19:13:50.886446 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.886320    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerDied","Data":"3fc18eb372dacda3e63ee22031b210b34d22aa93d60d724aae14a6fe34bcf0e2"}
Jul 10 19:13:50.887721 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.887698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerStarted","Data":"34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811"}
Jul 10 19:13:50.887831 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:50.887753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerStarted","Data":"b1a40ff3e4accdfd98cb32b25864d8efbb0b12cedfe6bbaadd0588d869806eaa"}
Jul 10 19:13:53.253027 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:53.252986    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 10 19:13:53.592974 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:53.592889    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Jul 10 19:13:54.903736 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:54.903708    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerID="34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811" exitCode=0
Jul 10 19:13:54.904139 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:54.903777    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerDied","Data":"34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811"}
Jul 10 19:13:54.906314 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:54.906296    2558 generic.go:356] "Generic (PLEG): container finished" podID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerID="5ef2fead58c11a4e041acc7280ecde26500eac0d136b34639bd50053f759619c" exitCode=0
Jul 10 19:13:54.906407 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:54.906328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerDied","Data":"5ef2fead58c11a4e041acc7280ecde26500eac0d136b34639bd50053f759619c"}
Jul 10 19:13:55.018006 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.017985    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:13:55.207637 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.207602    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnmtm\" (UniqueName: \"kubernetes.io/projected/17a4028a-2e82-4f38-95db-84f72a6cd286-kube-api-access-lnmtm\") pod \"17a4028a-2e82-4f38-95db-84f72a6cd286\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") "
Jul 10 19:13:55.207933 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.207915    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a4028a-2e82-4f38-95db-84f72a6cd286-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") pod \"17a4028a-2e82-4f38-95db-84f72a6cd286\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") "
Jul 10 19:13:55.208046 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.208033    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a4028a-2e82-4f38-95db-84f72a6cd286-kserve-provision-location\") pod \"17a4028a-2e82-4f38-95db-84f72a6cd286\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") "
Jul 10 19:13:55.208178 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.208163    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a4028a-2e82-4f38-95db-84f72a6cd286-proxy-tls\") pod \"17a4028a-2e82-4f38-95db-84f72a6cd286\" (UID: \"17a4028a-2e82-4f38-95db-84f72a6cd286\") "
Jul 10 19:13:55.208713 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.208676    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a4028a-2e82-4f38-95db-84f72a6cd286-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17a4028a-2e82-4f38-95db-84f72a6cd286" (UID: "17a4028a-2e82-4f38-95db-84f72a6cd286"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:13:55.208713 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.208700    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a4028a-2e82-4f38-95db-84f72a6cd286-isvc-predictive-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-kube-rbac-proxy-sar-config") pod "17a4028a-2e82-4f38-95db-84f72a6cd286" (UID: "17a4028a-2e82-4f38-95db-84f72a6cd286"). InnerVolumeSpecName "isvc-predictive-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:13:55.210479 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.210453    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a4028a-2e82-4f38-95db-84f72a6cd286-kube-api-access-lnmtm" (OuterVolumeSpecName: "kube-api-access-lnmtm") pod "17a4028a-2e82-4f38-95db-84f72a6cd286" (UID: "17a4028a-2e82-4f38-95db-84f72a6cd286"). InnerVolumeSpecName "kube-api-access-lnmtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:13:55.210580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.210559    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a4028a-2e82-4f38-95db-84f72a6cd286-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17a4028a-2e82-4f38-95db-84f72a6cd286" (UID: "17a4028a-2e82-4f38-95db-84f72a6cd286"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:13:55.309123 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.309099    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17a4028a-2e82-4f38-95db-84f72a6cd286-isvc-predictive-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:13:55.309123 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.309124    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17a4028a-2e82-4f38-95db-84f72a6cd286-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:13:55.309277 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.309134    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17a4028a-2e82-4f38-95db-84f72a6cd286-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:13:55.309277 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.309143    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnmtm\" (UniqueName: \"kubernetes.io/projected/17a4028a-2e82-4f38-95db-84f72a6cd286-kube-api-access-lnmtm\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:13:55.912268 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.912230    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerStarted","Data":"7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b"}
Jul 10 19:13:55.912709 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.912301    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerStarted","Data":"270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6"}
Jul 10 19:13:55.912709 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.912601    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:55.912709 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.912661    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:13:55.914239 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.914211    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2" event={"ID":"17a4028a-2e82-4f38-95db-84f72a6cd286","Type":"ContainerDied","Data":"f50fe8c1267527fe0f1911d492238220bb79e43ccc430a65b588b648d7f2515a"}
Jul 10 19:13:55.914346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.914264    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"
Jul 10 19:13:55.914346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.914276    2558 scope.go:117] "RemoveContainer" containerID="3fc18eb372dacda3e63ee22031b210b34d22aa93d60d724aae14a6fe34bcf0e2"
Jul 10 19:13:55.914346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.914251    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:13:55.922358 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.922341    2558 scope.go:117] "RemoveContainer" containerID="5ef2fead58c11a4e041acc7280ecde26500eac0d136b34639bd50053f759619c"
Jul 10 19:13:55.929037 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.929023    2558 scope.go:117] "RemoveContainer" containerID="2c04c020b539973e0d4e06be115dd44d184579152af5599b9a8496790a563fe5"
Jul 10 19:13:55.932010 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.931971    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podStartSLOduration=2.054031424 podStartE2EDuration="5.931958203s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877926779s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:13:55.930891274 +0000 UTC m=+2081.192556253" watchObservedRunningTime="2026-07-10 19:13:55.931958203 +0000 UTC m=+2081.193623056"
Jul 10 19:13:55.943531 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.943432    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"]
Jul 10 19:13:55.945060 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:55.945033    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-568c5d6bc7-9wxf2"]
Jul 10 19:13:56.918932 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:56.918894    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:13:57.257394 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:13:57.257361    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" path="/var/lib/kubelet/pods/17a4028a-2e82-4f38-95db-84f72a6cd286/volumes"
Jul 10 19:14:01.923638 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:01.923611    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:14:01.924175 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:01.924148    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:14:11.924459 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:11.924424    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:14:15.299491 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:15.299465    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:14:15.302395 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:15.302375    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:14:21.924933 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:21.924894    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:14:31.924262 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:31.924221    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:14:41.924255 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:41.924212    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:14:51.924223 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:14:51.924179    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:15:01.924641 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:01.924600    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:15:11.924986 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:11.924907    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:15:20.387823 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.387789    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"]
Jul 10 19:15:20.389352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.388069    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" containerID="cri-o://270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6" gracePeriod=30
Jul 10 19:15:20.389352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.388099    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kube-rbac-proxy" containerID="cri-o://7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b" gracePeriod=30
Jul 10 19:15:20.506872 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.506845    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"]
Jul 10 19:15:20.507111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507093    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="storage-initializer"
Jul 10 19:15:20.507111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507109    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="storage-initializer"
Jul 10 19:15:20.507348 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507124    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container"
Jul 10 19:15:20.507348 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507130    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container"
Jul 10 19:15:20.507348 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507149    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kube-rbac-proxy"
Jul 10 19:15:20.507348 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507157    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kube-rbac-proxy"
Jul 10 19:15:20.507348 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507208    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kube-rbac-proxy"
Jul 10 19:15:20.507348 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.507217    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17a4028a-2e82-4f38-95db-84f72a6cd286" containerName="kserve-container"
Jul 10 19:15:20.509111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.509095    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.511159 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.511138    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 10 19:15:20.511327 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.511308    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 10 19:15:20.524832 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.524807    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"]
Jul 10 19:15:20.604208 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.604181    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.604370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.604219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.604370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.604249    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxn87\" (UniqueName: \"kubernetes.io/projected/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kube-api-access-bxn87\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.604370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.604353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.705137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.705069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.705137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.705120    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.705299 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.705156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.705299 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.705201    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxn87\" (UniqueName: \"kubernetes.io/projected/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kube-api-access-bxn87\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.705509 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.705486    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.705889 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.705866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.707695 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.707676    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.713811 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.713793    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxn87\" (UniqueName: \"kubernetes.io/projected/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kube-api-access-bxn87\") pod \"isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.818992 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.818964    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:20.944162 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:20.944122    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"]
Jul 10 19:15:20.949196 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:15:20.947106    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53c30e8_c8ca_44b5_9fb5_845eb3c64be6.slice/crio-f92d8c27c89f012c7fff7842b4061319d988f8ed6e6abbbc17aed7abdcf8950b WatchSource:0}: Error finding container f92d8c27c89f012c7fff7842b4061319d988f8ed6e6abbbc17aed7abdcf8950b: Status 404 returned error can't find the container with id f92d8c27c89f012c7fff7842b4061319d988f8ed6e6abbbc17aed7abdcf8950b
Jul 10 19:15:21.217384 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:21.217288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerStarted","Data":"0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6"}
Jul 10 19:15:21.217384 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:21.217365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerStarted","Data":"f92d8c27c89f012c7fff7842b4061319d988f8ed6e6abbbc17aed7abdcf8950b"}
Jul 10 19:15:21.219521 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:21.219497    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerID="7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b" exitCode=2
Jul 10 19:15:21.219646 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:21.219572    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerDied","Data":"7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b"}
Jul 10 19:15:21.919299 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:21.919257    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 10 19:15:21.924639 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:21.924614    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 10 19:15:24.780822 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.780767    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:15:24.939111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.939075    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kserve-provision-location\") pod \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") "
Jul 10 19:15:24.939288 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.939142    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3ff6f07-2f9d-4985-ab5d-ee641349b567-proxy-tls\") pod \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") "
Jul 10 19:15:24.939288 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.939246    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4n9d\" (UniqueName: \"kubernetes.io/projected/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kube-api-access-q4n9d\") pod \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") "
Jul 10 19:15:24.939416 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.939302    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3ff6f07-2f9d-4985-ab5d-ee641349b567-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") pod \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\" (UID: \"f3ff6f07-2f9d-4985-ab5d-ee641349b567\") "
Jul 10 19:15:24.939514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.939455    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3ff6f07-2f9d-4985-ab5d-ee641349b567" (UID: "f3ff6f07-2f9d-4985-ab5d-ee641349b567"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:15:24.939626 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.939593    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ff6f07-2f9d-4985-ab5d-ee641349b567-isvc-predictive-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-kube-rbac-proxy-sar-config") pod "f3ff6f07-2f9d-4985-ab5d-ee641349b567" (UID: "f3ff6f07-2f9d-4985-ab5d-ee641349b567"). InnerVolumeSpecName "isvc-predictive-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:15:24.941175 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.941147    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ff6f07-2f9d-4985-ab5d-ee641349b567-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f3ff6f07-2f9d-4985-ab5d-ee641349b567" (UID: "f3ff6f07-2f9d-4985-ab5d-ee641349b567"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:15:24.941262 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:24.941187    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kube-api-access-q4n9d" (OuterVolumeSpecName: "kube-api-access-q4n9d") pod "f3ff6f07-2f9d-4985-ab5d-ee641349b567" (UID: "f3ff6f07-2f9d-4985-ab5d-ee641349b567"). InnerVolumeSpecName "kube-api-access-q4n9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:15:25.040090 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.040023    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:15:25.040090 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.040048    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f3ff6f07-2f9d-4985-ab5d-ee641349b567-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:15:25.040090 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.040058    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4n9d\" (UniqueName: \"kubernetes.io/projected/f3ff6f07-2f9d-4985-ab5d-ee641349b567-kube-api-access-q4n9d\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:15:25.040090 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.040067    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f3ff6f07-2f9d-4985-ab5d-ee641349b567-isvc-predictive-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:15:25.238272 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.238242    2558 generic.go:356] "Generic (PLEG): container finished" podID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerID="270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6" exitCode=0
Jul 10 19:15:25.238450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.238301    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"
Jul 10 19:15:25.238450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.238325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerDied","Data":"270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6"}
Jul 10 19:15:25.238450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.238391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc" event={"ID":"f3ff6f07-2f9d-4985-ab5d-ee641349b567","Type":"ContainerDied","Data":"b1a40ff3e4accdfd98cb32b25864d8efbb0b12cedfe6bbaadd0588d869806eaa"}
Jul 10 19:15:25.238450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.238429    2558 scope.go:117] "RemoveContainer" containerID="7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b"
Jul 10 19:15:25.240032 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.240007    2558 generic.go:356] "Generic (PLEG): container finished" podID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerID="0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6" exitCode=0
Jul 10 19:15:25.240165 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.240083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerDied","Data":"0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6"}
Jul 10 19:15:25.247212 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.247170    2558 scope.go:117] "RemoveContainer" containerID="270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6"
Jul 10 19:15:25.254757 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.254724    2558 scope.go:117] "RemoveContainer" containerID="34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811"
Jul 10 19:15:25.261860 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.261842    2558 scope.go:117] "RemoveContainer" containerID="7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b"
Jul 10 19:15:25.262128 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:15:25.262110    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b\": container with ID starting with 7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b not found: ID does not exist" containerID="7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b"
Jul 10 19:15:25.262177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.262134    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b"} err="failed to get container status \"7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b\": rpc error: code = NotFound desc = could not find container \"7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b\": container with ID starting with 7daadd815414ee4311e0b3a3a44a4891ac1a374221501ab02438950ac716ce0b not found: ID does not exist"
Jul 10 19:15:25.262177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.262150    2558 scope.go:117] "RemoveContainer" containerID="270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6"
Jul 10 19:15:25.262368 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:15:25.262352    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6\": container with ID starting with 270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6 not found: ID does not exist" containerID="270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6"
Jul 10 19:15:25.262436 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.262378    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6"} err="failed to get container status \"270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6\": rpc error: code = NotFound desc = could not find container \"270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6\": container with ID starting with 270f99a5c489db22c7e4ccb3c2bce14677f2c67c97d8b9f0a7b609b325109eb6 not found: ID does not exist"
Jul 10 19:15:25.262436 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.262402    2558 scope.go:117] "RemoveContainer" containerID="34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811"
Jul 10 19:15:25.262668 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:15:25.262646    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811\": container with ID starting with 34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811 not found: ID does not exist" containerID="34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811"
Jul 10 19:15:25.262725 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.262673    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811"} err="failed to get container status \"34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811\": rpc error: code = NotFound desc = could not find container \"34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811\": container with ID starting with 34bb6113792f5ba542c4ed1a5d2b7d338ab5df7fe886528aa770460a8c24a811 not found: ID does not exist"
Jul 10 19:15:25.280731 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.280702    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"]
Jul 10 19:15:25.286207 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:25.286187    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-f6d66fc5d-dfqjc"]
Jul 10 19:15:26.245715 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:26.245682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerStarted","Data":"5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86"}
Jul 10 19:15:26.246206 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:26.245758    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerStarted","Data":"9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd"}
Jul 10 19:15:26.246206 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:26.245965    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:26.264897 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:26.264850    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podStartSLOduration=2.546960052 podStartE2EDuration="6.264834986s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717874934s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:15:26.263443962 +0000 UTC m=+2171.525108734" watchObservedRunningTime="2026-07-10 19:15:26.264834986 +0000 UTC m=+2171.526499815"
Jul 10 19:15:27.250780 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:27.250737    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:27.252094 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:27.252067    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:15:27.256248 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:27.256222    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" path="/var/lib/kubelet/pods/f3ff6f07-2f9d-4985-ab5d-ee641349b567/volumes"
Jul 10 19:15:28.255134 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:28.255097    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:15:33.259263 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:33.259234    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:15:33.259869 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:33.259840    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:15:43.260257 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:43.260214    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:15:53.259956 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:15:53.259919    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:16:03.259801 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:03.259766    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:16:13.259863 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:13.259817    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:16:23.259896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:23.259854    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:16:33.260192 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:33.260148    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:16:43.260588 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:43.260518    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:16:50.969114 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969084    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"]
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969356    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kube-rbac-proxy"
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969366    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kube-rbac-proxy"
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969379    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="storage-initializer"
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969385    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="storage-initializer"
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969396    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container"
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969402    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container"
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969442    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kserve-container"
Jul 10 19:16:50.969545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.969449    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ff6f07-2f9d-4985-ab5d-ee641349b567" containerName="kube-rbac-proxy"
Jul 10 19:16:50.972283 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.972262    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:50.974557 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.974535    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 10 19:16:50.974774 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.974572    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 10 19:16:50.984258 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.984238    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"]
Jul 10 19:16:50.987386 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.987357    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"]
Jul 10 19:16:50.987590 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.987567    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" containerID="cri-o://9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd" gracePeriod=30
Jul 10 19:16:50.987710 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:50.987663    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kube-rbac-proxy" containerID="cri-o://5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86" gracePeriod=30
Jul 10 19:16:51.042058 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.042026    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/951444eb-efae-4f88-bc30-705d9976d939-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.042233 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.042075    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/951444eb-efae-4f88-bc30-705d9976d939-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.042233 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.042107    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.042233 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.042157    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv8zb\" (UniqueName: \"kubernetes.io/projected/951444eb-efae-4f88-bc30-705d9976d939-kube-api-access-mv8zb\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.143044 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.143004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/951444eb-efae-4f88-bc30-705d9976d939-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.143210 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.143053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.143255 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:16:51.143223    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 10 19:16:51.143309 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:16:51.143288    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls podName:951444eb-efae-4f88-bc30-705d9976d939 nodeName:}" failed. No retries permitted until 2026-07-10 19:16:51.643267268 +0000 UTC m=+2256.904932020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" (UID: "951444eb-efae-4f88-bc30-705d9976d939") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 10 19:16:51.143309 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.143303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv8zb\" (UniqueName: \"kubernetes.io/projected/951444eb-efae-4f88-bc30-705d9976d939-kube-api-access-mv8zb\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.143391 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.143329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/951444eb-efae-4f88-bc30-705d9976d939-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.143441 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.143399    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/951444eb-efae-4f88-bc30-705d9976d939-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.143944 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.143926    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/951444eb-efae-4f88-bc30-705d9976d939-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.151551 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.151528    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv8zb\" (UniqueName: \"kubernetes.io/projected/951444eb-efae-4f88-bc30-705d9976d939-kube-api-access-mv8zb\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.550870 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.550836    2558 generic.go:356] "Generic (PLEG): container finished" podID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerID="5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86" exitCode=2
Jul 10 19:16:51.551045 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.550911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerDied","Data":"5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86"}
Jul 10 19:16:51.646655 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.646621    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.649049 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.649030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:51.883855 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:51.883728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:52.012375 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:52.012350    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"]
Jul 10 19:16:52.014195 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:16:52.014166    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951444eb_efae_4f88_bc30_705d9976d939.slice/crio-dd3a038c14ff5699341d93f2976b5e714ffdf201c050439313ebba8d3e176683 WatchSource:0}: Error finding container dd3a038c14ff5699341d93f2976b5e714ffdf201c050439313ebba8d3e176683: Status 404 returned error can't find the container with id dd3a038c14ff5699341d93f2976b5e714ffdf201c050439313ebba8d3e176683
Jul 10 19:16:52.555863 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:52.555824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerStarted","Data":"74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54"}
Jul 10 19:16:52.556028 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:52.555886    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerStarted","Data":"dd3a038c14ff5699341d93f2976b5e714ffdf201c050439313ebba8d3e176683"}
Jul 10 19:16:53.255411 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:53.255374    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 10 19:16:53.260568 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:53.260542    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 10 19:16:55.423858 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.423830    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:16:55.567804 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.567775    2558 generic.go:356] "Generic (PLEG): container finished" podID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerID="9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd" exitCode=0
Jul 10 19:16:55.567952 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.567836    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerDied","Data":"9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd"}
Jul 10 19:16:55.567952 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.567897    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5" event={"ID":"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6","Type":"ContainerDied","Data":"f92d8c27c89f012c7fff7842b4061319d988f8ed6e6abbbc17aed7abdcf8950b"}
Jul 10 19:16:55.567952 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.567938    2558 scope.go:117] "RemoveContainer" containerID="5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86"
Jul 10 19:16:55.568059 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.567972    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"
Jul 10 19:16:55.575648 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.575624    2558 scope.go:117] "RemoveContainer" containerID="9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd"
Jul 10 19:16:55.580802 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.580776    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") pod \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") "
Jul 10 19:16:55.580909 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.580890    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxn87\" (UniqueName: \"kubernetes.io/projected/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kube-api-access-bxn87\") pod \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") "
Jul 10 19:16:55.580954 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.580929    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kserve-provision-location\") pod \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") "
Jul 10 19:16:55.580993 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.580962    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-proxy-tls\") pod \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\" (UID: \"a53c30e8-c8ca-44b5-9fb5-845eb3c64be6\") "
Jul 10 19:16:55.581211 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.581172    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config") pod "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" (UID: "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6"). InnerVolumeSpecName "isvc-predictive-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:16:55.581306 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.581218    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" (UID: "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:16:55.582896 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.582877    2558 scope.go:117] "RemoveContainer" containerID="0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6"
Jul 10 19:16:55.583088 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.583058    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" (UID: "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:16:55.583254 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.583224    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kube-api-access-bxn87" (OuterVolumeSpecName: "kube-api-access-bxn87") pod "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" (UID: "a53c30e8-c8ca-44b5-9fb5-845eb3c64be6"). InnerVolumeSpecName "kube-api-access-bxn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:16:55.596128 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.596112    2558 scope.go:117] "RemoveContainer" containerID="5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86"
Jul 10 19:16:55.596388 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:16:55.596370    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86\": container with ID starting with 5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86 not found: ID does not exist" containerID="5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86"
Jul 10 19:16:55.596448 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.596399    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86"} err="failed to get container status \"5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86\": rpc error: code = NotFound desc = could not find container \"5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86\": container with ID starting with 5b58c0665d4057ea5fd2f647d286b9bb6809943649c0b804d2e8a018aabdfc86 not found: ID does not exist"
Jul 10 19:16:55.596448 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.596417    2558 scope.go:117] "RemoveContainer" containerID="9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd"
Jul 10 19:16:55.596668 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:16:55.596650    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd\": container with ID starting with 9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd not found: ID does not exist" containerID="9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd"
Jul 10 19:16:55.596723 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.596674    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd"} err="failed to get container status \"9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd\": rpc error: code = NotFound desc = could not find container \"9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd\": container with ID starting with 9fdd86b71955cf650d738ff298004d6588ba6a767eb7055b83bacb61fbbee7dd not found: ID does not exist"
Jul 10 19:16:55.596723 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.596689    2558 scope.go:117] "RemoveContainer" containerID="0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6"
Jul 10 19:16:55.596962 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:16:55.596947    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6\": container with ID starting with 0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6 not found: ID does not exist" containerID="0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6"
Jul 10 19:16:55.597007 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.596966    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6"} err="failed to get container status \"0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6\": rpc error: code = NotFound desc = could not find container \"0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6\": container with ID starting with 0cb822b0f26da6534c6c4b2e541543bde6be72d965137d06921936a8801ab7b6 not found: ID does not exist"
Jul 10 19:16:55.681996 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.681963    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:16:55.681996 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.681994    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxn87\" (UniqueName: \"kubernetes.io/projected/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kube-api-access-bxn87\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:16:55.681996 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.682005    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:16:55.682170 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.682014    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:16:55.890489 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.890456    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"]
Jul 10 19:16:55.896238 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:55.896214    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-bbdb699d9-td9z5"]
Jul 10 19:16:56.572830 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:56.572800    2558 generic.go:356] "Generic (PLEG): container finished" podID="951444eb-efae-4f88-bc30-705d9976d939" containerID="74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54" exitCode=0
Jul 10 19:16:56.573345 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:56.572873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerDied","Data":"74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54"}
Jul 10 19:16:57.256370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:57.256337    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" path="/var/lib/kubelet/pods/a53c30e8-c8ca-44b5-9fb5-845eb3c64be6/volumes"
Jul 10 19:16:57.579637 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:57.579558    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerStarted","Data":"80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9"}
Jul 10 19:16:57.579637 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:57.579618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerStarted","Data":"5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717"}
Jul 10 19:16:57.580051 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:57.579807    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:16:57.580051 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:16:57.579852    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:17:03.589077 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:17:03.589047    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:17:03.610787 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:17:03.610717    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podStartSLOduration=10.061792292 podStartE2EDuration="13.610702245s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.548909953s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:16:57.602691717 +0000 UTC m=+2262.864356529" watchObservedRunningTime="2026-07-10 19:17:03.610702245 +0000 UTC m=+2268.872367018"
Jul 10 19:17:33.590420 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:17:33.590379    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 10 19:17:43.590032 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:17:43.589986    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 10 19:17:53.590425 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:17:53.590341    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 10 19:18:03.589988 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:03.589947    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 10 19:18:14.509369 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:14.509289    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:18:21.152683 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.152652    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"]
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153054    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container"
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153072    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container"
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153098    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="storage-initializer"
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153107    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="storage-initializer"
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153125    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kube-rbac-proxy"
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153134    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kube-rbac-proxy"
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153200    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kserve-container"
Jul 10 19:18:21.153280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.153211    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53c30e8-c8ca-44b5-9fb5-845eb3c64be6" containerName="kube-rbac-proxy"
Jul 10 19:18:21.156463 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.156440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.157969 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.157946    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"]
Jul 10 19:18:21.158194 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.158150    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container" containerID="cri-o://5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717" gracePeriod=30
Jul 10 19:18:21.158297 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.158249    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kube-rbac-proxy" containerID="cri-o://80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9" gracePeriod=30
Jul 10 19:18:21.158920 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.158902    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 10 19:18:21.158976 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.158905    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 10 19:18:21.166126 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.166104    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"]
Jul 10 19:18:21.273971 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.273945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396cd025-93fd-4314-a643-9adc056a3baf-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.274116 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.273993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396cd025-93fd-4314-a643-9adc056a3baf-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.274116 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.274025    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvq2\" (UniqueName: \"kubernetes.io/projected/396cd025-93fd-4314-a643-9adc056a3baf-kube-api-access-2lvq2\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.274116 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.274044    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396cd025-93fd-4314-a643-9adc056a3baf-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.374483 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.374456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396cd025-93fd-4314-a643-9adc056a3baf-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.374618 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.374491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvq2\" (UniqueName: \"kubernetes.io/projected/396cd025-93fd-4314-a643-9adc056a3baf-kube-api-access-2lvq2\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.374618 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.374511    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396cd025-93fd-4314-a643-9adc056a3baf-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.374618 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.374550    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396cd025-93fd-4314-a643-9adc056a3baf-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.374901 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.374885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396cd025-93fd-4314-a643-9adc056a3baf-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.375165 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.375147    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396cd025-93fd-4314-a643-9adc056a3baf-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.376795 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.376769    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396cd025-93fd-4314-a643-9adc056a3baf-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.383186 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.383165    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvq2\" (UniqueName: \"kubernetes.io/projected/396cd025-93fd-4314-a643-9adc056a3baf-kube-api-access-2lvq2\") pod \"isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.474537 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.474475    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:21.602942 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.602908    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"]
Jul 10 19:18:21.605477 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:18:21.605450    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396cd025_93fd_4314_a643_9adc056a3baf.slice/crio-4c8ee8462497421eb7d2f8a7c0cc9e6f392c41ee39dbde3415e3dc0930c6f00c WatchSource:0}: Error finding container 4c8ee8462497421eb7d2f8a7c0cc9e6f392c41ee39dbde3415e3dc0930c6f00c: Status 404 returned error can't find the container with id 4c8ee8462497421eb7d2f8a7c0cc9e6f392c41ee39dbde3415e3dc0930c6f00c
Jul 10 19:18:21.879092 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.879055    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerStarted","Data":"292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824"}
Jul 10 19:18:21.879270 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.879124    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerStarted","Data":"4c8ee8462497421eb7d2f8a7c0cc9e6f392c41ee39dbde3415e3dc0930c6f00c"}
Jul 10 19:18:21.881401 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.881375    2558 generic.go:356] "Generic (PLEG): container finished" podID="951444eb-efae-4f88-bc30-705d9976d939" containerID="80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9" exitCode=2
Jul 10 19:18:21.881513 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:21.881412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerDied","Data":"80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9"}
Jul 10 19:18:23.584492 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:23.584450    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Jul 10 19:18:23.589961 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:23.589934    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.42:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.42:8080: connect: connection refused"
Jul 10 19:18:25.404090 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.404065    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:18:25.503603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.503579    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/951444eb-efae-4f88-bc30-705d9976d939-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"951444eb-efae-4f88-bc30-705d9976d939\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") "
Jul 10 19:18:25.503772 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.503634    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls\") pod \"951444eb-efae-4f88-bc30-705d9976d939\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") "
Jul 10 19:18:25.503772 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.503757    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv8zb\" (UniqueName: \"kubernetes.io/projected/951444eb-efae-4f88-bc30-705d9976d939-kube-api-access-mv8zb\") pod \"951444eb-efae-4f88-bc30-705d9976d939\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") "
Jul 10 19:18:25.503902 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.503793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/951444eb-efae-4f88-bc30-705d9976d939-kserve-provision-location\") pod \"951444eb-efae-4f88-bc30-705d9976d939\" (UID: \"951444eb-efae-4f88-bc30-705d9976d939\") "
Jul 10 19:18:25.503993 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.503967    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951444eb-efae-4f88-bc30-705d9976d939-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config") pod "951444eb-efae-4f88-bc30-705d9976d939" (UID: "951444eb-efae-4f88-bc30-705d9976d939"). InnerVolumeSpecName "isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:18:25.504138 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.504118    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/951444eb-efae-4f88-bc30-705d9976d939-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "951444eb-efae-4f88-bc30-705d9976d939" (UID: "951444eb-efae-4f88-bc30-705d9976d939"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:18:25.505552 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.505528    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "951444eb-efae-4f88-bc30-705d9976d939" (UID: "951444eb-efae-4f88-bc30-705d9976d939"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:18:25.505700 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.505645    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951444eb-efae-4f88-bc30-705d9976d939-kube-api-access-mv8zb" (OuterVolumeSpecName: "kube-api-access-mv8zb") pod "951444eb-efae-4f88-bc30-705d9976d939" (UID: "951444eb-efae-4f88-bc30-705d9976d939"). InnerVolumeSpecName "kube-api-access-mv8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:18:25.604470 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.604436    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/951444eb-efae-4f88-bc30-705d9976d939-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:18:25.604470 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.604468    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv8zb\" (UniqueName: \"kubernetes.io/projected/951444eb-efae-4f88-bc30-705d9976d939-kube-api-access-mv8zb\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:18:25.604470 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.604477    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/951444eb-efae-4f88-bc30-705d9976d939-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:18:25.604637 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.604489    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/951444eb-efae-4f88-bc30-705d9976d939-isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:18:25.898164 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.898082    2558 generic.go:356] "Generic (PLEG): container finished" podID="951444eb-efae-4f88-bc30-705d9976d939" containerID="5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717" exitCode=0
Jul 10 19:18:25.898164 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.898147    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"
Jul 10 19:18:25.898380 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.898159    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerDied","Data":"5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717"}
Jul 10 19:18:25.898380 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.898217    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d" event={"ID":"951444eb-efae-4f88-bc30-705d9976d939","Type":"ContainerDied","Data":"dd3a038c14ff5699341d93f2976b5e714ffdf201c050439313ebba8d3e176683"}
Jul 10 19:18:25.898380 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.898247    2558 scope.go:117] "RemoveContainer" containerID="80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9"
Jul 10 19:18:25.899901 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.899871    2558 generic.go:356] "Generic (PLEG): container finished" podID="396cd025-93fd-4314-a643-9adc056a3baf" containerID="292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824" exitCode=0
Jul 10 19:18:25.900020 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.899933    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerDied","Data":"292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824"}
Jul 10 19:18:25.906569 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.906448    2558 scope.go:117] "RemoveContainer" containerID="5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717"
Jul 10 19:18:25.914158 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.914141    2558 scope.go:117] "RemoveContainer" containerID="74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54"
Jul 10 19:18:25.922909 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.922892    2558 scope.go:117] "RemoveContainer" containerID="80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9"
Jul 10 19:18:25.923150 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:18:25.923127    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9\": container with ID starting with 80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9 not found: ID does not exist" containerID="80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9"
Jul 10 19:18:25.923228 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.923162    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9"} err="failed to get container status \"80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9\": rpc error: code = NotFound desc = could not find container \"80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9\": container with ID starting with 80b7fe8750b050502b0df06226356d54e0dec4cac67d7014142bbea55e8f78f9 not found: ID does not exist"
Jul 10 19:18:25.923228 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.923187    2558 scope.go:117] "RemoveContainer" containerID="5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717"
Jul 10 19:18:25.923478 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:18:25.923452    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717\": container with ID starting with 5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717 not found: ID does not exist" containerID="5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717"
Jul 10 19:18:25.923539 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.923486    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717"} err="failed to get container status \"5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717\": rpc error: code = NotFound desc = could not find container \"5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717\": container with ID starting with 5398ca72e860996d8376fa01f6d849ea28631f72bf9b8b252a10095dfa456717 not found: ID does not exist"
Jul 10 19:18:25.923539 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.923507    2558 scope.go:117] "RemoveContainer" containerID="74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54"
Jul 10 19:18:25.923825 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:18:25.923808    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54\": container with ID starting with 74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54 not found: ID does not exist" containerID="74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54"
Jul 10 19:18:25.923874 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.923829    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54"} err="failed to get container status \"74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54\": rpc error: code = NotFound desc = could not find container \"74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54\": container with ID starting with 74ed61cb3059aa7604f4c3c8c523ab071afbc264f8c69b33d4e899521b862a54 not found: ID does not exist"
Jul 10 19:18:25.932272 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.932252    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"]
Jul 10 19:18:25.936053 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:25.936034    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-689589cc95-4w45d"]
Jul 10 19:18:26.906073 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:26.906041    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerStarted","Data":"280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a"}
Jul 10 19:18:26.906519 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:26.906113    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerStarted","Data":"c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963"}
Jul 10 19:18:26.906519 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:26.906327    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:26.926076 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:26.926031    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podStartSLOduration=2.182045293 podStartE2EDuration="5.926016546s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.743971253s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:18:26.923872014 +0000 UTC m=+2352.185536787" watchObservedRunningTime="2026-07-10 19:18:26.926016546 +0000 UTC m=+2352.187681318"
Jul 10 19:18:27.256637 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:27.256605    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951444eb-efae-4f88-bc30-705d9976d939" path="/var/lib/kubelet/pods/951444eb-efae-4f88-bc30-705d9976d939/volumes"
Jul 10 19:18:27.911507 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:27.911481    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:18:33.920405 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:18:33.920377    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:19:03.921802 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:03.921764    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 10 19:19:13.921213 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:13.921173    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 10 19:19:15.321134 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:15.321107    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:19:15.324117 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:15.324096    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:19:23.921036 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:23.920993    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 10 19:19:33.921823 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:33.921784    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 10 19:19:35.307155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:35.307123    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:19:41.246106 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.246028    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"]
Jul 10 19:19:41.246466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.246277    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container" containerID="cri-o://c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963" gracePeriod=30
Jul 10 19:19:41.246466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.246367    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kube-rbac-proxy" containerID="cri-o://280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a" gracePeriod=30
Jul 10 19:19:41.346407 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346361    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"]
Jul 10 19:19:41.346634 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346621    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container"
Jul 10 19:19:41.346681 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346636    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container"
Jul 10 19:19:41.346681 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346647    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kube-rbac-proxy"
Jul 10 19:19:41.346681 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346653    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kube-rbac-proxy"
Jul 10 19:19:41.346681 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346662    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="storage-initializer"
Jul 10 19:19:41.346681 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346668    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="storage-initializer"
Jul 10 19:19:41.346879 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346720    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kserve-container"
Jul 10 19:19:41.346879 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.346728    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="951444eb-efae-4f88-bc30-705d9976d939" containerName="kube-rbac-proxy"
Jul 10 19:19:41.349577 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.349558    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.352459 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.352436    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 10 19:19:41.352879 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.352859    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 10 19:19:41.363993 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.363971    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"]
Jul 10 19:19:41.449587 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.449539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fde7340-315d-415e-80e0-f24bf1abcde9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.449587 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.449593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nqq\" (UniqueName: \"kubernetes.io/projected/3fde7340-315d-415e-80e0-f24bf1abcde9-kube-api-access-s7nqq\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.449843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.449648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fde7340-315d-415e-80e0-f24bf1abcde9-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.449843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.449709    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fde7340-315d-415e-80e0-f24bf1abcde9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.550459 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.550387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fde7340-315d-415e-80e0-f24bf1abcde9-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.550459 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.550424    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fde7340-315d-415e-80e0-f24bf1abcde9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.550459 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.550455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fde7340-315d-415e-80e0-f24bf1abcde9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.550706 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.550478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nqq\" (UniqueName: \"kubernetes.io/projected/3fde7340-315d-415e-80e0-f24bf1abcde9-kube-api-access-s7nqq\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.550899 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.550875    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fde7340-315d-415e-80e0-f24bf1abcde9-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.551374 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.551351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fde7340-315d-415e-80e0-f24bf1abcde9-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.552811 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.552792    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fde7340-315d-415e-80e0-f24bf1abcde9-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.560584 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.560557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nqq\" (UniqueName: \"kubernetes.io/projected/3fde7340-315d-415e-80e0-f24bf1abcde9-kube-api-access-s7nqq\") pod \"isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.660222 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.660194    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:41.787830 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:41.787800    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"]
Jul 10 19:19:41.790661 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:19:41.790633    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fde7340_315d_415e_80e0_f24bf1abcde9.slice/crio-f462c37d1e553767a04fb3715814542a6cea1c838d03571e3ab08db945f37d60 WatchSource:0}: Error finding container f462c37d1e553767a04fb3715814542a6cea1c838d03571e3ab08db945f37d60: Status 404 returned error can't find the container with id f462c37d1e553767a04fb3715814542a6cea1c838d03571e3ab08db945f37d60
Jul 10 19:19:42.169983 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:42.169887    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerStarted","Data":"a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906"}
Jul 10 19:19:42.169983 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:42.169950    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerStarted","Data":"f462c37d1e553767a04fb3715814542a6cea1c838d03571e3ab08db945f37d60"}
Jul 10 19:19:42.171924 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:42.171899    2558 generic.go:356] "Generic (PLEG): container finished" podID="396cd025-93fd-4314-a643-9adc056a3baf" containerID="280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a" exitCode=2
Jul 10 19:19:42.172042 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:42.171968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerDied","Data":"280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a"}
Jul 10 19:19:43.916014 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:43.915975    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Jul 10 19:19:45.256895 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.256861    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Jul 10 19:19:45.482006 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.481983    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:19:45.578614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.578535    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lvq2\" (UniqueName: \"kubernetes.io/projected/396cd025-93fd-4314-a643-9adc056a3baf-kube-api-access-2lvq2\") pod \"396cd025-93fd-4314-a643-9adc056a3baf\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") "
Jul 10 19:19:45.578614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.578575    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396cd025-93fd-4314-a643-9adc056a3baf-kserve-provision-location\") pod \"396cd025-93fd-4314-a643-9adc056a3baf\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") "
Jul 10 19:19:45.578850 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.578614    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396cd025-93fd-4314-a643-9adc056a3baf-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"396cd025-93fd-4314-a643-9adc056a3baf\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") "
Jul 10 19:19:45.578850 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.578659    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396cd025-93fd-4314-a643-9adc056a3baf-proxy-tls\") pod \"396cd025-93fd-4314-a643-9adc056a3baf\" (UID: \"396cd025-93fd-4314-a643-9adc056a3baf\") "
Jul 10 19:19:45.578982 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.578945    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/396cd025-93fd-4314-a643-9adc056a3baf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "396cd025-93fd-4314-a643-9adc056a3baf" (UID: "396cd025-93fd-4314-a643-9adc056a3baf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:19:45.579085 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.578992    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396cd025-93fd-4314-a643-9adc056a3baf-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config") pod "396cd025-93fd-4314-a643-9adc056a3baf" (UID: "396cd025-93fd-4314-a643-9adc056a3baf"). InnerVolumeSpecName "isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:19:45.580620 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.580596    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396cd025-93fd-4314-a643-9adc056a3baf-kube-api-access-2lvq2" (OuterVolumeSpecName: "kube-api-access-2lvq2") pod "396cd025-93fd-4314-a643-9adc056a3baf" (UID: "396cd025-93fd-4314-a643-9adc056a3baf"). InnerVolumeSpecName "kube-api-access-2lvq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:19:45.580697 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.580633    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396cd025-93fd-4314-a643-9adc056a3baf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "396cd025-93fd-4314-a643-9adc056a3baf" (UID: "396cd025-93fd-4314-a643-9adc056a3baf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:19:45.679649 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.679622    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/396cd025-93fd-4314-a643-9adc056a3baf-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:19:45.679649 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.679642    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvq2\" (UniqueName: \"kubernetes.io/projected/396cd025-93fd-4314-a643-9adc056a3baf-kube-api-access-2lvq2\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:19:45.679649 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.679652    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/396cd025-93fd-4314-a643-9adc056a3baf-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:19:45.679860 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:45.679661    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/396cd025-93fd-4314-a643-9adc056a3baf-isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:19:46.189087 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.189009    2558 generic.go:356] "Generic (PLEG): container finished" podID="396cd025-93fd-4314-a643-9adc056a3baf" containerID="c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963" exitCode=0
Jul 10 19:19:46.189087 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.189060    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"
Jul 10 19:19:46.189275 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.189092    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerDied","Data":"c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963"}
Jul 10 19:19:46.189275 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.189154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8" event={"ID":"396cd025-93fd-4314-a643-9adc056a3baf","Type":"ContainerDied","Data":"4c8ee8462497421eb7d2f8a7c0cc9e6f392c41ee39dbde3415e3dc0930c6f00c"}
Jul 10 19:19:46.189275 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.189190    2558 scope.go:117] "RemoveContainer" containerID="280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a"
Jul 10 19:19:46.190782 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.190758    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerID="a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906" exitCode=0
Jul 10 19:19:46.190906 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.190787    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerDied","Data":"a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906"}
Jul 10 19:19:46.197500 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.197480    2558 scope.go:117] "RemoveContainer" containerID="c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963"
Jul 10 19:19:46.205620 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.205599    2558 scope.go:117] "RemoveContainer" containerID="292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824"
Jul 10 19:19:46.216377 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.216359    2558 scope.go:117] "RemoveContainer" containerID="280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a"
Jul 10 19:19:46.216605 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:19:46.216589    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a\": container with ID starting with 280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a not found: ID does not exist" containerID="280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a"
Jul 10 19:19:46.216686 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.216612    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a"} err="failed to get container status \"280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a\": rpc error: code = NotFound desc = could not find container \"280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a\": container with ID starting with 280cf6dc7a0629e3e4680466a74c4486918a60193985c601f2ad5bc4d4d5fd2a not found: ID does not exist"
Jul 10 19:19:46.216686 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.216628    2558 scope.go:117] "RemoveContainer" containerID="c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963"
Jul 10 19:19:46.216848 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:19:46.216835    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963\": container with ID starting with c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963 not found: ID does not exist" containerID="c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963"
Jul 10 19:19:46.216906 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.216852    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963"} err="failed to get container status \"c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963\": rpc error: code = NotFound desc = could not find container \"c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963\": container with ID starting with c9ede3934c629c141fb278bd0428f8cf2aa7f2b92f6be2f5114283ff5fbf4963 not found: ID does not exist"
Jul 10 19:19:46.216906 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.216865    2558 scope.go:117] "RemoveContainer" containerID="292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824"
Jul 10 19:19:46.217078 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:19:46.217061    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824\": container with ID starting with 292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824 not found: ID does not exist" containerID="292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824"
Jul 10 19:19:46.217146 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.217080    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824"} err="failed to get container status \"292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824\": rpc error: code = NotFound desc = could not find container \"292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824\": container with ID starting with 292ad57be735a140a3ac4167cde37636a4e696e8f42d3dac123e9cc4fcfb3824 not found: ID does not exist"
Jul 10 19:19:46.256200 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.256181    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"]
Jul 10 19:19:46.259272 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:46.259251    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-64bc9f8f76-477z8"]
Jul 10 19:19:47.197241 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:47.197203    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerStarted","Data":"1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474"}
Jul 10 19:19:47.197423 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:47.197261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerStarted","Data":"103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b"}
Jul 10 19:19:47.197511 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:47.197494    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:47.220929 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:47.220884    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podStartSLOduration=2.199708438 podStartE2EDuration="6.220869663s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.021161225s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:19:47.219134908 +0000 UTC m=+2432.480799680" watchObservedRunningTime="2026-07-10 19:19:47.220869663 +0000 UTC m=+2432.482534435"
Jul 10 19:19:47.257091 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:47.257064    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="396cd025-93fd-4314-a643-9adc056a3baf" path="/var/lib/kubelet/pods/396cd025-93fd-4314-a643-9adc056a3baf/volumes"
Jul 10 19:19:48.201256 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:48.201220    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:19:54.213870 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:19:54.213834    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:20:24.214714 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:20:24.214675    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 10 19:20:34.214507 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:20:34.214461    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 10 19:20:44.215224 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:20:44.215181    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 10 19:20:54.214638 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:20:54.214592    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Jul 10 19:20:57.307016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:20:57.306981    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:21:01.613648 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:01.613617    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"]
Jul 10 19:21:01.614129 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:01.613907    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container" containerID="cri-o://103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b" gracePeriod=30
Jul 10 19:21:01.614129 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:01.613975    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kube-rbac-proxy" containerID="cri-o://1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474" gracePeriod=30
Jul 10 19:21:02.467106 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:02.467070    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerID="1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474" exitCode=2
Jul 10 19:21:02.467275 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:02.467146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerDied","Data":"1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474"}
Jul 10 19:21:03.639873 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.639840    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"]
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640084    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kube-rbac-proxy"
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640095    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kube-rbac-proxy"
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640107    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container"
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640115    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container"
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640129    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="storage-initializer"
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640135    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="storage-initializer"
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640182    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kube-rbac-proxy"
Jul 10 19:21:03.640232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.640189    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="396cd025-93fd-4314-a643-9adc056a3baf" containerName="kserve-container"
Jul 10 19:21:03.642954 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.642938    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.645246 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.645223    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 10 19:21:03.645359 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.645330    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 10 19:21:03.658213 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.658193    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"]
Jul 10 19:21:03.739728 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.739702    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-proxy-tls\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.739899 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.739762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kserve-provision-location\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.739899 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.739864    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq474\" (UniqueName: \"kubernetes.io/projected/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kube-api-access-rq474\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.740017 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.739913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.840973 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.840948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kserve-provision-location\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.841111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.840995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq474\" (UniqueName: \"kubernetes.io/projected/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kube-api-access-rq474\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.841111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.841026    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.841111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.841058    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-proxy-tls\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.841397 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.841378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kserve-provision-location\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.841760 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.841719    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.843407 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.843382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-proxy-tls\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.850289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.850266    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq474\" (UniqueName: \"kubernetes.io/projected/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kube-api-access-rq474\") pod \"isvc-sklearn-predictor-7cc4654bd-r5qpp\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:03.953402 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:03.953326    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:04.093394 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:04.093358    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"]
Jul 10 19:21:04.096330 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:21:04.096305    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158f1e38_e2d6_4a25_a11c_0921bbfb2f60.slice/crio-3044e0bdb18dcbb6896f13d99c5b943073400c706b6b7b018f0af39abba86667 WatchSource:0}: Error finding container 3044e0bdb18dcbb6896f13d99c5b943073400c706b6b7b018f0af39abba86667: Status 404 returned error can't find the container with id 3044e0bdb18dcbb6896f13d99c5b943073400c706b6b7b018f0af39abba86667
Jul 10 19:21:04.204978 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:04.204898    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Jul 10 19:21:04.478282 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:04.478190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerStarted","Data":"806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e"}
Jul 10 19:21:04.478282 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:04.478255    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerStarted","Data":"3044e0bdb18dcbb6896f13d99c5b943073400c706b6b7b018f0af39abba86667"}
Jul 10 19:21:06.253867 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.253829    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:21:06.361187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.361151    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fde7340-315d-415e-80e0-f24bf1abcde9-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") pod \"3fde7340-315d-415e-80e0-f24bf1abcde9\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") "
Jul 10 19:21:06.361355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.361203    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7nqq\" (UniqueName: \"kubernetes.io/projected/3fde7340-315d-415e-80e0-f24bf1abcde9-kube-api-access-s7nqq\") pod \"3fde7340-315d-415e-80e0-f24bf1abcde9\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") "
Jul 10 19:21:06.361355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.361241    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fde7340-315d-415e-80e0-f24bf1abcde9-proxy-tls\") pod \"3fde7340-315d-415e-80e0-f24bf1abcde9\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") "
Jul 10 19:21:06.361355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.361275    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fde7340-315d-415e-80e0-f24bf1abcde9-kserve-provision-location\") pod \"3fde7340-315d-415e-80e0-f24bf1abcde9\" (UID: \"3fde7340-315d-415e-80e0-f24bf1abcde9\") "
Jul 10 19:21:06.361551 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.361523    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fde7340-315d-415e-80e0-f24bf1abcde9-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config") pod "3fde7340-315d-415e-80e0-f24bf1abcde9" (UID: "3fde7340-315d-415e-80e0-f24bf1abcde9"). InnerVolumeSpecName "isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:21:06.361717 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.361694    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fde7340-315d-415e-80e0-f24bf1abcde9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3fde7340-315d-415e-80e0-f24bf1abcde9" (UID: "3fde7340-315d-415e-80e0-f24bf1abcde9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:21:06.363343 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.363321    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fde7340-315d-415e-80e0-f24bf1abcde9-kube-api-access-s7nqq" (OuterVolumeSpecName: "kube-api-access-s7nqq") pod "3fde7340-315d-415e-80e0-f24bf1abcde9" (UID: "3fde7340-315d-415e-80e0-f24bf1abcde9"). InnerVolumeSpecName "kube-api-access-s7nqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:21:06.363343 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.363329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fde7340-315d-415e-80e0-f24bf1abcde9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3fde7340-315d-415e-80e0-f24bf1abcde9" (UID: "3fde7340-315d-415e-80e0-f24bf1abcde9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:21:06.462016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.461937    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nqq\" (UniqueName: \"kubernetes.io/projected/3fde7340-315d-415e-80e0-f24bf1abcde9-kube-api-access-s7nqq\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:21:06.462016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.461964    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fde7340-315d-415e-80e0-f24bf1abcde9-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:21:06.462016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.461973    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fde7340-315d-415e-80e0-f24bf1abcde9-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:21:06.462016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.461982    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fde7340-315d-415e-80e0-f24bf1abcde9-isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:21:06.487563 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.487535    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerID="103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b" exitCode=0
Jul 10 19:21:06.487679 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.487581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerDied","Data":"103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b"}
Jul 10 19:21:06.487679 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.487591    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"
Jul 10 19:21:06.487679 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.487627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln" event={"ID":"3fde7340-315d-415e-80e0-f24bf1abcde9","Type":"ContainerDied","Data":"f462c37d1e553767a04fb3715814542a6cea1c838d03571e3ab08db945f37d60"}
Jul 10 19:21:06.487679 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.487653    2558 scope.go:117] "RemoveContainer" containerID="1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474"
Jul 10 19:21:06.495990 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.495969    2558 scope.go:117] "RemoveContainer" containerID="103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b"
Jul 10 19:21:06.503122 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.503104    2558 scope.go:117] "RemoveContainer" containerID="a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906"
Jul 10 19:21:06.508363 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.508334    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"]
Jul 10 19:21:06.510510 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.510491    2558 scope.go:117] "RemoveContainer" containerID="1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474"
Jul 10 19:21:06.510786 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:21:06.510763    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474\": container with ID starting with 1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474 not found: ID does not exist" containerID="1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474"
Jul 10 19:21:06.510843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.510797    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474"} err="failed to get container status \"1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474\": rpc error: code = NotFound desc = could not find container \"1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474\": container with ID starting with 1e03aa1a15ce784ec37a1e578a3948574bf113bd6f045f0e6792d0979809f474 not found: ID does not exist"
Jul 10 19:21:06.510843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.510816    2558 scope.go:117] "RemoveContainer" containerID="103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b"
Jul 10 19:21:06.511062 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:21:06.511044    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b\": container with ID starting with 103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b not found: ID does not exist" containerID="103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b"
Jul 10 19:21:06.511103 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.511069    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b"} err="failed to get container status \"103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b\": rpc error: code = NotFound desc = could not find container \"103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b\": container with ID starting with 103fa1f9d3177a6f4c6f937b575e24e06855783c9aeaedfeaa3365144576475b not found: ID does not exist"
Jul 10 19:21:06.511103 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.511086    2558 scope.go:117] "RemoveContainer" containerID="a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906"
Jul 10 19:21:06.511317 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:21:06.511300    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906\": container with ID starting with a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906 not found: ID does not exist" containerID="a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906"
Jul 10 19:21:06.511380 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.511320    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906"} err="failed to get container status \"a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906\": rpc error: code = NotFound desc = could not find container \"a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906\": container with ID starting with a8d98a3db37b1019667a02e8d717baf5037ac7df63a7e256e774deb804372906 not found: ID does not exist"
Jul 10 19:21:06.513006 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:06.512987    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-5f7f654675-85nln"]
Jul 10 19:21:07.257506 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:07.257471    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" path="/var/lib/kubelet/pods/3fde7340-315d-415e-80e0-f24bf1abcde9/volumes"
Jul 10 19:21:08.501970 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:08.501940    2558 generic.go:356] "Generic (PLEG): container finished" podID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerID="806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e" exitCode=0
Jul 10 19:21:08.502387 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:08.502005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerDied","Data":"806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e"}
Jul 10 19:21:09.508196 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:09.508164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerStarted","Data":"c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313"}
Jul 10 19:21:09.508542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:09.508232    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerStarted","Data":"510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3"}
Jul 10 19:21:09.508542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:09.508423    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:09.531905 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:09.531861    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podStartSLOduration=2.84468838 podStartE2EDuration="6.531848281s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.687159901s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:21:09.530257716 +0000 UTC m=+2514.791922488" watchObservedRunningTime="2026-07-10 19:21:09.531848281 +0000 UTC m=+2514.793513049"
Jul 10 19:21:10.512260 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:10.512226    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:10.513510 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:10.513473    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:21:11.515738 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:11.515696    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:21:16.519887 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:16.519855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:21:16.520363 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:16.520337    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:21:26.520517 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:26.520471    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:21:36.520537 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:36.520498    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:21:46.521194 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:46.521154    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:21:56.520640 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:21:56.520601    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:22:06.520642 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:06.520606    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:22:16.520904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:16.520874    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:22:23.754085 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.754054    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"]
Jul 10 19:22:23.754506 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.754275    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" containerID="cri-o://510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3" gracePeriod=30
Jul 10 19:22:23.754506 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.754329    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kube-rbac-proxy" containerID="cri-o://c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313" gracePeriod=30
Jul 10 19:22:23.855860 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.855834    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"]
Jul 10 19:22:23.856123 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856110    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="storage-initializer"
Jul 10 19:22:23.856170 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856125    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="storage-initializer"
Jul 10 19:22:23.856170 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856132    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container"
Jul 10 19:22:23.856170 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856138    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container"
Jul 10 19:22:23.856170 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856145    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kube-rbac-proxy"
Jul 10 19:22:23.856170 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856151    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kube-rbac-proxy"
Jul 10 19:22:23.856325 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856193    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kube-rbac-proxy"
Jul 10 19:22:23.856325 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.856200    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fde7340-315d-415e-80e0-f24bf1abcde9" containerName="kserve-container"
Jul 10 19:22:23.859547 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.859525    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:23.861922 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.861889    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 10 19:22:23.861922 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.861911    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 10 19:22:23.870938 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.870921    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"]
Jul 10 19:22:23.919605 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.919577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c37c05-966e-4584-bbb2-bd1c483f7dea-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:23.919775 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.919619    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c37c05-966e-4584-bbb2-bd1c483f7dea-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:23.919775 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.919689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c37c05-966e-4584-bbb2-bd1c483f7dea-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:23.919775 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:23.919724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2fp8\" (UniqueName: \"kubernetes.io/projected/68c37c05-966e-4584-bbb2-bd1c483f7dea-kube-api-access-c2fp8\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.025765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.020768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c37c05-966e-4584-bbb2-bd1c483f7dea-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.025765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.020896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c37c05-966e-4584-bbb2-bd1c483f7dea-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.025765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.020979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c37c05-966e-4584-bbb2-bd1c483f7dea-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.025765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.021032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2fp8\" (UniqueName: \"kubernetes.io/projected/68c37c05-966e-4584-bbb2-bd1c483f7dea-kube-api-access-c2fp8\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.025765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.021157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c37c05-966e-4584-bbb2-bd1c483f7dea-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.025765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.022127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c37c05-966e-4584-bbb2-bd1c483f7dea-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.027154 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.027128    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c37c05-966e-4584-bbb2-bd1c483f7dea-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.031280 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.031259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2fp8\" (UniqueName: \"kubernetes.io/projected/68c37c05-966e-4584-bbb2-bd1c483f7dea-kube-api-access-c2fp8\") pod \"sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.172168 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.172131    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:24.300802 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.300656    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"]
Jul 10 19:22:24.303236 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:22:24.303205    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c37c05_966e_4584_bbb2_bd1c483f7dea.slice/crio-a01d600534cea11da62a234db31b81d049950dc76eb639a9c220217f540ca1f2 WatchSource:0}: Error finding container a01d600534cea11da62a234db31b81d049950dc76eb639a9c220217f540ca1f2: Status 404 returned error can't find the container with id a01d600534cea11da62a234db31b81d049950dc76eb639a9c220217f540ca1f2
Jul 10 19:22:24.771867 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.771834    2558 generic.go:356] "Generic (PLEG): container finished" podID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerID="c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313" exitCode=2
Jul 10 19:22:24.772236 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.771909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerDied","Data":"c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313"}
Jul 10 19:22:24.773505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.773484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerStarted","Data":"2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba"}
Jul 10 19:22:24.773632 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:24.773536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerStarted","Data":"a01d600534cea11da62a234db31b81d049950dc76eb639a9c220217f540ca1f2"}
Jul 10 19:22:26.516283 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:26.516244    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Jul 10 19:22:26.520523 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:26.520499    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Jul 10 19:22:27.696780 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.696757    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:22:27.746318 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.746280    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-proxy-tls\") pod \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") "
Jul 10 19:22:27.746466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.746357    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") "
Jul 10 19:22:27.746466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.746382    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq474\" (UniqueName: \"kubernetes.io/projected/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kube-api-access-rq474\") pod \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") "
Jul 10 19:22:27.746466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.746408    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kserve-provision-location\") pod \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\" (UID: \"158f1e38-e2d6-4a25-a11c-0921bbfb2f60\") "
Jul 10 19:22:27.746726 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.746704    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "158f1e38-e2d6-4a25-a11c-0921bbfb2f60" (UID: "158f1e38-e2d6-4a25-a11c-0921bbfb2f60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:22:27.746816 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.746724    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "158f1e38-e2d6-4a25-a11c-0921bbfb2f60" (UID: "158f1e38-e2d6-4a25-a11c-0921bbfb2f60"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:22:27.748434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.748408    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "158f1e38-e2d6-4a25-a11c-0921bbfb2f60" (UID: "158f1e38-e2d6-4a25-a11c-0921bbfb2f60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:22:27.748434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.748429    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kube-api-access-rq474" (OuterVolumeSpecName: "kube-api-access-rq474") pod "158f1e38-e2d6-4a25-a11c-0921bbfb2f60" (UID: "158f1e38-e2d6-4a25-a11c-0921bbfb2f60"). InnerVolumeSpecName "kube-api-access-rq474". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:22:27.785971 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.785909    2558 generic.go:356] "Generic (PLEG): container finished" podID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerID="510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3" exitCode=0
Jul 10 19:22:27.785971 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.785967    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"
Jul 10 19:22:27.786120 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.785971    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerDied","Data":"510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3"}
Jul 10 19:22:27.786120 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.786019    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp" event={"ID":"158f1e38-e2d6-4a25-a11c-0921bbfb2f60","Type":"ContainerDied","Data":"3044e0bdb18dcbb6896f13d99c5b943073400c706b6b7b018f0af39abba86667"}
Jul 10 19:22:27.786120 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.786045    2558 scope.go:117] "RemoveContainer" containerID="c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313"
Jul 10 19:22:27.793906 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.793890    2558 scope.go:117] "RemoveContainer" containerID="510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3"
Jul 10 19:22:27.803537 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.803517    2558 scope.go:117] "RemoveContainer" containerID="806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e"
Jul 10 19:22:27.806496 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.806473    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"]
Jul 10 19:22:27.811725 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.811704    2558 scope.go:117] "RemoveContainer" containerID="c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313"
Jul 10 19:22:27.812042 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:22:27.812021    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313\": container with ID starting with c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313 not found: ID does not exist" containerID="c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313"
Jul 10 19:22:27.812125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.812056    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313"} err="failed to get container status \"c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313\": rpc error: code = NotFound desc = could not find container \"c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313\": container with ID starting with c04f04696530773e8f3530cd7016b2dad576b6f310932042cc92c53deeea8313 not found: ID does not exist"
Jul 10 19:22:27.812125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.812082    2558 scope.go:117] "RemoveContainer" containerID="510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3"
Jul 10 19:22:27.812278 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.812259    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-7cc4654bd-r5qpp"]
Jul 10 19:22:27.812438 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:22:27.812420    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3\": container with ID starting with 510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3 not found: ID does not exist" containerID="510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3"
Jul 10 19:22:27.812523 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.812447    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3"} err="failed to get container status \"510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3\": rpc error: code = NotFound desc = could not find container \"510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3\": container with ID starting with 510f6e1903c1afb0a0af59f4d94040fcbd0e07685fbe5f6b480637faf9d6cdc3 not found: ID does not exist"
Jul 10 19:22:27.812523 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.812469    2558 scope.go:117] "RemoveContainer" containerID="806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e"
Jul 10 19:22:27.812852 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:22:27.812831    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e\": container with ID starting with 806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e not found: ID does not exist" containerID="806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e"
Jul 10 19:22:27.812915 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.812859    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e"} err="failed to get container status \"806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e\": rpc error: code = NotFound desc = could not find container \"806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e\": container with ID starting with 806abae518df807e375d0691cd7feefc8e3c791ae89e222debd96fc48c37a69e not found: ID does not exist"
Jul 10 19:22:27.847027 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.847002    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:22:27.847119 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.847031    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq474\" (UniqueName: \"kubernetes.io/projected/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kube-api-access-rq474\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:22:27.847119 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.847047    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:22:27.847119 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:27.847059    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/158f1e38-e2d6-4a25-a11c-0921bbfb2f60-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:22:28.797725 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:28.797643    2558 generic.go:356] "Generic (PLEG): container finished" podID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerID="2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba" exitCode=0
Jul 10 19:22:28.797725 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:28.797695    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerDied","Data":"2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba"}
Jul 10 19:22:29.256452 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:29.256422    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" path="/var/lib/kubelet/pods/158f1e38-e2d6-4a25-a11c-0921bbfb2f60/volumes"
Jul 10 19:22:29.802752 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:29.802719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerStarted","Data":"9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46"}
Jul 10 19:22:29.803217 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:29.802806    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerStarted","Data":"7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a"}
Jul 10 19:22:29.803217 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:29.803037    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:29.803217 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:29.803082    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:22:29.823761 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:29.823703    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" podStartSLOduration=3.332662868 podStartE2EDuration="6.823689656s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.491026788s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:22:29.82240858 +0000 UTC m=+2595.084073364" watchObservedRunningTime="2026-07-10 19:22:29.823689656 +0000 UTC m=+2595.085354428"
Jul 10 19:22:35.812925 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:22:35.812896    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:23:05.903691 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:05.903638    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 10 19:23:15.815882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:15.815845    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:23:23.941371 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:23.941341    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"]
Jul 10 19:23:23.941818 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:23.941553    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kserve-container" containerID="cri-o://7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a" gracePeriod=30
Jul 10 19:23:23.941818 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:23.941598    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kube-rbac-proxy" containerID="cri-o://9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46" gracePeriod=30
Jul 10 19:23:24.002948 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.002915    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"]
Jul 10 19:23:24.003203 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003190    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container"
Jul 10 19:23:24.003257 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003204    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container"
Jul 10 19:23:24.003257 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003218    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="storage-initializer"
Jul 10 19:23:24.003257 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003224    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="storage-initializer"
Jul 10 19:23:24.003257 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003231    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kube-rbac-proxy"
Jul 10 19:23:24.003257 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003237    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kube-rbac-proxy"
Jul 10 19:23:24.003432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003287    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kube-rbac-proxy"
Jul 10 19:23:24.003432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.003301    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="158f1e38-e2d6-4a25-a11c-0921bbfb2f60" containerName="kserve-container"
Jul 10 19:23:24.007600 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.007579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.010339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.010318    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:23:24.010456 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.010318    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 10 19:23:24.016997 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.016946    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"]
Jul 10 19:23:24.142938 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.142909    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nkp\" (UniqueName: \"kubernetes.io/projected/2645f280-a226-4082-a12d-737b0dd0a9a5-kube-api-access-d4nkp\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.143074 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.142948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2645f280-a226-4082-a12d-737b0dd0a9a5-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.143074 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.142971    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2645f280-a226-4082-a12d-737b0dd0a9a5-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.143074 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.143042    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2645f280-a226-4082-a12d-737b0dd0a9a5-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.244189 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.244166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nkp\" (UniqueName: \"kubernetes.io/projected/2645f280-a226-4082-a12d-737b0dd0a9a5-kube-api-access-d4nkp\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.244309 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.244209    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2645f280-a226-4082-a12d-737b0dd0a9a5-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.244309 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.244248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2645f280-a226-4082-a12d-737b0dd0a9a5-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.244309 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.244288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2645f280-a226-4082-a12d-737b0dd0a9a5-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.244792 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.244763    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2645f280-a226-4082-a12d-737b0dd0a9a5-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.244942 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.244925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2645f280-a226-4082-a12d-737b0dd0a9a5-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.246595 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.246578    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2645f280-a226-4082-a12d-737b0dd0a9a5-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.252214 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.252194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nkp\" (UniqueName: \"kubernetes.io/projected/2645f280-a226-4082-a12d-737b0dd0a9a5-kube-api-access-d4nkp\") pod \"isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.318857 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.318837    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:24.446505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.446475    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"]
Jul 10 19:23:24.448725 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:23:24.448699    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2645f280_a226_4082_a12d_737b0dd0a9a5.slice/crio-a389d7c056209f6299be4a8c0707b64ad53bb1724e751f86181e051715019472 WatchSource:0}: Error finding container a389d7c056209f6299be4a8c0707b64ad53bb1724e751f86181e051715019472: Status 404 returned error can't find the container with id a389d7c056209f6299be4a8c0707b64ad53bb1724e751f86181e051715019472
Jul 10 19:23:24.994202 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.994172    2558 generic.go:356] "Generic (PLEG): container finished" podID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerID="9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46" exitCode=2
Jul 10 19:23:24.994621 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.994248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerDied","Data":"9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46"}
Jul 10 19:23:24.995550 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.995524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerStarted","Data":"79dfd49be1ea6307a2877cf9c04f50042469a8ec9796a28120eba96106a8c3a9"}
Jul 10 19:23:24.995656 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:24.995585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerStarted","Data":"a389d7c056209f6299be4a8c0707b64ad53bb1724e751f86181e051715019472"}
Jul 10 19:23:25.806906 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:25.806868    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 10 19:23:26.854931 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:26.854882    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.46:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 10 19:23:30.807010 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:30.806965    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Jul 10 19:23:30.989943 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:30.989917    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:23:31.021371 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.021339    2558 generic.go:356] "Generic (PLEG): container finished" podID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerID="7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a" exitCode=0
Jul 10 19:23:31.021520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.021405    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"
Jul 10 19:23:31.021520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.021408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerDied","Data":"7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a"}
Jul 10 19:23:31.021520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.021479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr" event={"ID":"68c37c05-966e-4584-bbb2-bd1c483f7dea","Type":"ContainerDied","Data":"a01d600534cea11da62a234db31b81d049950dc76eb639a9c220217f540ca1f2"}
Jul 10 19:23:31.021520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.021496    2558 scope.go:117] "RemoveContainer" containerID="9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46"
Jul 10 19:23:31.022960 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.022936    2558 generic.go:356] "Generic (PLEG): container finished" podID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerID="79dfd49be1ea6307a2877cf9c04f50042469a8ec9796a28120eba96106a8c3a9" exitCode=0
Jul 10 19:23:31.023064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.022972    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerDied","Data":"79dfd49be1ea6307a2877cf9c04f50042469a8ec9796a28120eba96106a8c3a9"}
Jul 10 19:23:31.030853 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.030830    2558 scope.go:117] "RemoveContainer" containerID="7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a"
Jul 10 19:23:31.038313 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.038295    2558 scope.go:117] "RemoveContainer" containerID="2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba"
Jul 10 19:23:31.049125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.049104    2558 scope.go:117] "RemoveContainer" containerID="9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46"
Jul 10 19:23:31.049370 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:23:31.049351    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46\": container with ID starting with 9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46 not found: ID does not exist" containerID="9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46"
Jul 10 19:23:31.049435 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.049378    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46"} err="failed to get container status \"9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46\": rpc error: code = NotFound desc = could not find container \"9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46\": container with ID starting with 9fcb35a9214076986100da1ae5b0a7576cd02d4b12e28e06007a29981c735f46 not found: ID does not exist"
Jul 10 19:23:31.049435 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.049395    2558 scope.go:117] "RemoveContainer" containerID="7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a"
Jul 10 19:23:31.049634 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:23:31.049616    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a\": container with ID starting with 7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a not found: ID does not exist" containerID="7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a"
Jul 10 19:23:31.049699 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.049639    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a"} err="failed to get container status \"7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a\": rpc error: code = NotFound desc = could not find container \"7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a\": container with ID starting with 7e7eae49e1552dbc9211510300912fa859695137bfb0e88ae6b10323ba42018a not found: ID does not exist"
Jul 10 19:23:31.049699 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.049653    2558 scope.go:117] "RemoveContainer" containerID="2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba"
Jul 10 19:23:31.049956 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:23:31.049938    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba\": container with ID starting with 2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba not found: ID does not exist" containerID="2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba"
Jul 10 19:23:31.050033 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.049962    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba"} err="failed to get container status \"2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba\": rpc error: code = NotFound desc = could not find container \"2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba\": container with ID starting with 2bd6a1fb09513237500087d430d640e7492c6a70c5c9a032b281262abcf544ba not found: ID does not exist"
Jul 10 19:23:31.100970 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.100944    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c37c05-966e-4584-bbb2-bd1c483f7dea-kserve-provision-location\") pod \"68c37c05-966e-4584-bbb2-bd1c483f7dea\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") "
Jul 10 19:23:31.101097 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.100988    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c37c05-966e-4584-bbb2-bd1c483f7dea-proxy-tls\") pod \"68c37c05-966e-4584-bbb2-bd1c483f7dea\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") "
Jul 10 19:23:31.101097 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.101071    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c37c05-966e-4584-bbb2-bd1c483f7dea-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"68c37c05-966e-4584-bbb2-bd1c483f7dea\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") "
Jul 10 19:23:31.101215 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.101105    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2fp8\" (UniqueName: \"kubernetes.io/projected/68c37c05-966e-4584-bbb2-bd1c483f7dea-kube-api-access-c2fp8\") pod \"68c37c05-966e-4584-bbb2-bd1c483f7dea\" (UID: \"68c37c05-966e-4584-bbb2-bd1c483f7dea\") "
Jul 10 19:23:31.101764 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.101467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c37c05-966e-4584-bbb2-bd1c483f7dea-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "68c37c05-966e-4584-bbb2-bd1c483f7dea" (UID: "68c37c05-966e-4584-bbb2-bd1c483f7dea"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:23:31.101764 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.101641    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c37c05-966e-4584-bbb2-bd1c483f7dea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68c37c05-966e-4584-bbb2-bd1c483f7dea" (UID: "68c37c05-966e-4584-bbb2-bd1c483f7dea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:23:31.103429 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.103407    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c37c05-966e-4584-bbb2-bd1c483f7dea-kube-api-access-c2fp8" (OuterVolumeSpecName: "kube-api-access-c2fp8") pod "68c37c05-966e-4584-bbb2-bd1c483f7dea" (UID: "68c37c05-966e-4584-bbb2-bd1c483f7dea"). InnerVolumeSpecName "kube-api-access-c2fp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:23:31.103622 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.103595    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c37c05-966e-4584-bbb2-bd1c483f7dea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68c37c05-966e-4584-bbb2-bd1c483f7dea" (UID: "68c37c05-966e-4584-bbb2-bd1c483f7dea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:23:31.201627 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.201599    2558 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68c37c05-966e-4584-bbb2-bd1c483f7dea-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:23:31.201627 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.201622    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2fp8\" (UniqueName: \"kubernetes.io/projected/68c37c05-966e-4584-bbb2-bd1c483f7dea-kube-api-access-c2fp8\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:23:31.201627 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.201632    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68c37c05-966e-4584-bbb2-bd1c483f7dea-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:23:31.201936 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.201642    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68c37c05-966e-4584-bbb2-bd1c483f7dea-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:23:31.337617 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.337543    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"]
Jul 10 19:23:31.342302 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:31.342274    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7dcddcfbb4-ntdkr"]
Jul 10 19:23:32.031146 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:32.031111    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerStarted","Data":"a79ac8e25929e14eaca7e41d4f53828f27b7648cbc7749d052fc31621c9281c4"}
Jul 10 19:23:32.031547 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:32.031194    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerStarted","Data":"70624f0fce9bf10c38f777db723f79cbdf917bbb03aca094d3c23be1747f8bcf"}
Jul 10 19:23:32.031547 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:32.031437    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:32.050541 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:32.050491    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" podStartSLOduration=3.506417372 podStartE2EDuration="9.050475682s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.54405831s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:23:32.04871855 +0000 UTC m=+2657.310383335" watchObservedRunningTime="2026-07-10 19:23:32.050475682 +0000 UTC m=+2657.312140455"
Jul 10 19:23:33.035926 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:33.035887    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:33.036961 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:33.036935    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 10 19:23:33.257131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:33.257100    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" path="/var/lib/kubelet/pods/68c37c05-966e-4584-bbb2-bd1c483f7dea/volumes"
Jul 10 19:23:34.040065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:34.040023    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 10 19:23:39.044253 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:39.044229    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:23:39.047999 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:39.047937    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Jul 10 19:23:49.045903 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:23:49.045870    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:24:01.049443 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.049410    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq_2645f280-a226-4082-a12d-737b0dd0a9a5/kserve-container/0.log"
Jul 10 19:24:01.195080 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.195045    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"]
Jul 10 19:24:01.195375 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.195345    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kserve-container" containerID="cri-o://70624f0fce9bf10c38f777db723f79cbdf917bbb03aca094d3c23be1747f8bcf" gracePeriod=30
Jul 10 19:24:01.195470 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.195422    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kube-rbac-proxy" containerID="cri-o://a79ac8e25929e14eaca7e41d4f53828f27b7648cbc7749d052fc31621c9281c4" gracePeriod=30
Jul 10 19:24:01.276371 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276338    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"]
Jul 10 19:24:01.276596 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276584    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kube-rbac-proxy"
Jul 10 19:24:01.276641 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276598    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kube-rbac-proxy"
Jul 10 19:24:01.276641 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276609    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kserve-container"
Jul 10 19:24:01.276641 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276615    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kserve-container"
Jul 10 19:24:01.276641 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276627    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="storage-initializer"
Jul 10 19:24:01.276641 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276634    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="storage-initializer"
Jul 10 19:24:01.276829 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276682    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kserve-container"
Jul 10 19:24:01.276829 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.276690    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c37c05-966e-4584-bbb2-bd1c483f7dea" containerName="kube-rbac-proxy"
Jul 10 19:24:01.278945 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.278925    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.281760 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.281724    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 10 19:24:01.282071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.282051    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:24:01.290476 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.290455    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"]
Jul 10 19:24:01.324817 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.324789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45321d3c-e503-4bee-8254-94317035f5ce-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.324944 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.324838    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhzl\" (UniqueName: \"kubernetes.io/projected/45321d3c-e503-4bee-8254-94317035f5ce-kube-api-access-4hhzl\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.324944 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.324890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45321d3c-e503-4bee-8254-94317035f5ce-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.325048 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.324974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.425338 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.425313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhzl\" (UniqueName: \"kubernetes.io/projected/45321d3c-e503-4bee-8254-94317035f5ce-kube-api-access-4hhzl\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.425490 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.425343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45321d3c-e503-4bee-8254-94317035f5ce-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.425490 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.425388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.425490 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.425412    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45321d3c-e503-4bee-8254-94317035f5ce-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.425655 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:24:01.425537    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 10 19:24:01.425655 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:24:01.425603    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls podName:45321d3c-e503-4bee-8254-94317035f5ce nodeName:}" failed. No retries permitted until 2026-07-10 19:24:01.925582761 +0000 UTC m=+2687.187247513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" (UID: "45321d3c-e503-4bee-8254-94317035f5ce") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 10 19:24:01.425848 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.425829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45321d3c-e503-4bee-8254-94317035f5ce-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.426111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.426094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45321d3c-e503-4bee-8254-94317035f5ce-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.434405 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.434381    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhzl\" (UniqueName: \"kubernetes.io/projected/45321d3c-e503-4bee-8254-94317035f5ce-kube-api-access-4hhzl\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.929244 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.929200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:01.931626 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:01.931600    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:02.139087 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.139054    2558 generic.go:356] "Generic (PLEG): container finished" podID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerID="a79ac8e25929e14eaca7e41d4f53828f27b7648cbc7749d052fc31621c9281c4" exitCode=2
Jul 10 19:24:02.139087 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.139079    2558 generic.go:356] "Generic (PLEG): container finished" podID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerID="70624f0fce9bf10c38f777db723f79cbdf917bbb03aca094d3c23be1747f8bcf" exitCode=0
Jul 10 19:24:02.139447 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.139109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerDied","Data":"a79ac8e25929e14eaca7e41d4f53828f27b7648cbc7749d052fc31621c9281c4"}
Jul 10 19:24:02.139447 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.139157    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerDied","Data":"70624f0fce9bf10c38f777db723f79cbdf917bbb03aca094d3c23be1747f8bcf"}
Jul 10 19:24:02.190086 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.190022    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:02.247205 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.247177    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:24:02.323678 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.323652    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"]
Jul 10 19:24:02.332238 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.332215    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2645f280-a226-4082-a12d-737b0dd0a9a5-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"2645f280-a226-4082-a12d-737b0dd0a9a5\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") "
Jul 10 19:24:02.332354 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.332261    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2645f280-a226-4082-a12d-737b0dd0a9a5-kserve-provision-location\") pod \"2645f280-a226-4082-a12d-737b0dd0a9a5\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") "
Jul 10 19:24:02.332472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.332379    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4nkp\" (UniqueName: \"kubernetes.io/projected/2645f280-a226-4082-a12d-737b0dd0a9a5-kube-api-access-d4nkp\") pod \"2645f280-a226-4082-a12d-737b0dd0a9a5\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") "
Jul 10 19:24:02.332472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.332443    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2645f280-a226-4082-a12d-737b0dd0a9a5-proxy-tls\") pod \"2645f280-a226-4082-a12d-737b0dd0a9a5\" (UID: \"2645f280-a226-4082-a12d-737b0dd0a9a5\") "
Jul 10 19:24:02.332689 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.332533    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2645f280-a226-4082-a12d-737b0dd0a9a5-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "2645f280-a226-4082-a12d-737b0dd0a9a5" (UID: "2645f280-a226-4082-a12d-737b0dd0a9a5"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:24:02.332689 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.332628    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2645f280-a226-4082-a12d-737b0dd0a9a5-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:24:02.334376 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.334352    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2645f280-a226-4082-a12d-737b0dd0a9a5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2645f280-a226-4082-a12d-737b0dd0a9a5" (UID: "2645f280-a226-4082-a12d-737b0dd0a9a5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:24:02.334556 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.334537    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2645f280-a226-4082-a12d-737b0dd0a9a5-kube-api-access-d4nkp" (OuterVolumeSpecName: "kube-api-access-d4nkp") pod "2645f280-a226-4082-a12d-737b0dd0a9a5" (UID: "2645f280-a226-4082-a12d-737b0dd0a9a5"). InnerVolumeSpecName "kube-api-access-d4nkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:24:02.348802 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.348778    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2645f280-a226-4082-a12d-737b0dd0a9a5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2645f280-a226-4082-a12d-737b0dd0a9a5" (UID: "2645f280-a226-4082-a12d-737b0dd0a9a5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:24:02.432946 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.432919    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nkp\" (UniqueName: \"kubernetes.io/projected/2645f280-a226-4082-a12d-737b0dd0a9a5-kube-api-access-d4nkp\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:24:02.432946 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.432942    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2645f280-a226-4082-a12d-737b0dd0a9a5-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:24:02.433088 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:02.432952    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2645f280-a226-4082-a12d-737b0dd0a9a5-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:24:03.144848 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.144821    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"
Jul 10 19:24:03.145279 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.144820    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq" event={"ID":"2645f280-a226-4082-a12d-737b0dd0a9a5","Type":"ContainerDied","Data":"a389d7c056209f6299be4a8c0707b64ad53bb1724e751f86181e051715019472"}
Jul 10 19:24:03.145279 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.144966    2558 scope.go:117] "RemoveContainer" containerID="a79ac8e25929e14eaca7e41d4f53828f27b7648cbc7749d052fc31621c9281c4"
Jul 10 19:24:03.146471 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.146443    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerStarted","Data":"e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37"}
Jul 10 19:24:03.146567 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.146508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerStarted","Data":"05dcd2dd81fd03f81907ae5acb2ca06965a269d5d4948598bbf06df5997366de"}
Jul 10 19:24:03.153720 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.153702    2558 scope.go:117] "RemoveContainer" containerID="70624f0fce9bf10c38f777db723f79cbdf917bbb03aca094d3c23be1747f8bcf"
Jul 10 19:24:03.160568 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.160549    2558 scope.go:117] "RemoveContainer" containerID="79dfd49be1ea6307a2877cf9c04f50042469a8ec9796a28120eba96106a8c3a9"
Jul 10 19:24:03.180707 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.180683    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"]
Jul 10 19:24:03.186668 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.186648    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-75676ff7cd-cjbwq"]
Jul 10 19:24:03.256727 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:03.256704    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" path="/var/lib/kubelet/pods/2645f280-a226-4082-a12d-737b0dd0a9a5/volumes"
Jul 10 19:24:07.166649 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:07.166610    2558 generic.go:356] "Generic (PLEG): container finished" podID="45321d3c-e503-4bee-8254-94317035f5ce" containerID="e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37" exitCode=0
Jul 10 19:24:07.167054 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:07.166686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerDied","Data":"e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37"}
Jul 10 19:24:08.172360 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:08.172290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerStarted","Data":"97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785"}
Jul 10 19:24:08.172719 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:08.172381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerStarted","Data":"0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3"}
Jul 10 19:24:08.172719 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:08.172590    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:08.197379 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:08.197335    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" podStartSLOduration=3.422956188 podStartE2EDuration="7.197320419s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.774364231s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:24:08.196203217 +0000 UTC m=+2693.457868007" watchObservedRunningTime="2026-07-10 19:24:08.197320419 +0000 UTC m=+2693.458985250"
Jul 10 19:24:09.175558 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:09.175531    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:15.184767 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:15.184726    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:24:15.340878 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:15.340851    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:24:15.346938 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:15.346918    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:24:45.202128 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:45.202084    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 10 19:24:55.187527 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:24:55.187495    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:25:01.421147 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421110    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"]
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421377    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kserve-container"
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421389    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kserve-container"
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421406    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="storage-initializer"
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421413    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="storage-initializer"
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421419    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kube-rbac-proxy"
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421425    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kube-rbac-proxy"
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421472    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kube-rbac-proxy"
Jul 10 19:25:01.421514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.421481    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2645f280-a226-4082-a12d-737b0dd0a9a5" containerName="kserve-container"
Jul 10 19:25:01.424457 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.424432    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.426924 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.426902    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 10 19:25:01.427139 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.427110    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 10 19:25:01.436655 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.436624    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"]
Jul 10 19:25:01.450081 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.450056    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"]
Jul 10 19:25:01.450306 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.450282    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kserve-container" containerID="cri-o://0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3" gracePeriod=30
Jul 10 19:25:01.450380 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.450345    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kube-rbac-proxy" containerID="cri-o://97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785" gracePeriod=30
Jul 10 19:25:01.539542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.539510    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5c739-279a-4a09-9686-2acac54f14a1-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.539722 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.539550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.539722 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.539584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5c739-279a-4a09-9686-2acac54f14a1-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.539722 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.539636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjv9\" (UniqueName: \"kubernetes.io/projected/4ba5c739-279a-4a09-9686-2acac54f14a1-kube-api-access-9xjv9\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.640193 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.640154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5c739-279a-4a09-9686-2acac54f14a1-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.640193 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.640198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.640434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.640242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5c739-279a-4a09-9686-2acac54f14a1-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.640434 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:25:01.640382    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 10 19:25:01.640548 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.640435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjv9\" (UniqueName: \"kubernetes.io/projected/4ba5c739-279a-4a09-9686-2acac54f14a1-kube-api-access-9xjv9\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.640548 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:25:01.640465    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls podName:4ba5c739-279a-4a09-9686-2acac54f14a1 nodeName:}" failed. No retries permitted until 2026-07-10 19:25:02.140448046 +0000 UTC m=+2747.402112800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls") pod "isvc-sklearn-v2-predictor-66758b956b-5nz8j" (UID: "4ba5c739-279a-4a09-9686-2acac54f14a1") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Jul 10 19:25:01.640671 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.640658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5c739-279a-4a09-9686-2acac54f14a1-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.640937 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.640918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5c739-279a-4a09-9686-2acac54f14a1-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:01.650995 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:01.650972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjv9\" (UniqueName: \"kubernetes.io/projected/4ba5c739-279a-4a09-9686-2acac54f14a1-kube-api-access-9xjv9\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:02.144484 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:02.144448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:02.146842 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:02.146813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls\") pod \"isvc-sklearn-v2-predictor-66758b956b-5nz8j\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:02.336975 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:02.336938    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:02.362886 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:02.362856    2558 generic.go:356] "Generic (PLEG): container finished" podID="45321d3c-e503-4bee-8254-94317035f5ce" containerID="97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785" exitCode=2
Jul 10 19:25:02.363023 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:02.362911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerDied","Data":"97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785"}
Jul 10 19:25:02.472455 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:02.472210    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"]
Jul 10 19:25:02.474931 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:25:02.474906    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba5c739_279a_4a09_9686_2acac54f14a1.slice/crio-277f72011b9c0a1ec0ef331e803b2afda7d6f3ba4a7c6bf30a000eb9c80ced28 WatchSource:0}: Error finding container 277f72011b9c0a1ec0ef331e803b2afda7d6f3ba4a7c6bf30a000eb9c80ced28: Status 404 returned error can't find the container with id 277f72011b9c0a1ec0ef331e803b2afda7d6f3ba4a7c6bf30a000eb9c80ced28
Jul 10 19:25:03.368243 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:03.368203    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerStarted","Data":"a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e"}
Jul 10 19:25:03.368420 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:03.368264    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerStarted","Data":"277f72011b9c0a1ec0ef331e803b2afda7d6f3ba4a7c6bf30a000eb9c80ced28"}
Jul 10 19:25:05.179461 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:05.179423    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Jul 10 19:25:05.185523 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:05.185497    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.132.0.48:8080: connect: connection refused"
Jul 10 19:25:06.379472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:06.379440    2558 generic.go:356] "Generic (PLEG): container finished" podID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerID="a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e" exitCode=0
Jul 10 19:25:06.379876 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:06.379492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerDied","Data":"a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e"}
Jul 10 19:25:07.384709 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:07.384679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerStarted","Data":"c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b"}
Jul 10 19:25:07.385130 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:07.384761    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerStarted","Data":"1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101"}
Jul 10 19:25:07.385130 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:07.385035    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:07.385229 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:07.385183    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:07.386400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:07.386374    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:25:07.405071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:07.405018    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podStartSLOduration=2.797216852 podStartE2EDuration="6.405002165s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.607785313s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:25:07.40360025 +0000 UTC m=+2752.665265026" watchObservedRunningTime="2026-07-10 19:25:07.405002165 +0000 UTC m=+2752.666666938"
Jul 10 19:25:08.388408 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.388370    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:25:08.596149 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.596125    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:25:08.692952 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.692885    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls\") pod \"45321d3c-e503-4bee-8254-94317035f5ce\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") "
Jul 10 19:25:08.692952 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.692923    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45321d3c-e503-4bee-8254-94317035f5ce-kserve-provision-location\") pod \"45321d3c-e503-4bee-8254-94317035f5ce\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") "
Jul 10 19:25:08.693113 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.692969    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45321d3c-e503-4bee-8254-94317035f5ce-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"45321d3c-e503-4bee-8254-94317035f5ce\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") "
Jul 10 19:25:08.693113 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.693012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hhzl\" (UniqueName: \"kubernetes.io/projected/45321d3c-e503-4bee-8254-94317035f5ce-kube-api-access-4hhzl\") pod \"45321d3c-e503-4bee-8254-94317035f5ce\" (UID: \"45321d3c-e503-4bee-8254-94317035f5ce\") "
Jul 10 19:25:08.693252 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.693179    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45321d3c-e503-4bee-8254-94317035f5ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45321d3c-e503-4bee-8254-94317035f5ce" (UID: "45321d3c-e503-4bee-8254-94317035f5ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:25:08.693366 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.693342    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45321d3c-e503-4bee-8254-94317035f5ce-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "45321d3c-e503-4bee-8254-94317035f5ce" (UID: "45321d3c-e503-4bee-8254-94317035f5ce"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:25:08.694948 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.694923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45321d3c-e503-4bee-8254-94317035f5ce" (UID: "45321d3c-e503-4bee-8254-94317035f5ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:25:08.695060 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.694969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45321d3c-e503-4bee-8254-94317035f5ce-kube-api-access-4hhzl" (OuterVolumeSpecName: "kube-api-access-4hhzl") pod "45321d3c-e503-4bee-8254-94317035f5ce" (UID: "45321d3c-e503-4bee-8254-94317035f5ce"). InnerVolumeSpecName "kube-api-access-4hhzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:25:08.794472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.794443    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45321d3c-e503-4bee-8254-94317035f5ce-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:25:08.794472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.794470    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhzl\" (UniqueName: \"kubernetes.io/projected/45321d3c-e503-4bee-8254-94317035f5ce-kube-api-access-4hhzl\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:25:08.794472 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.794481    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45321d3c-e503-4bee-8254-94317035f5ce-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:25:08.794692 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:08.794490    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45321d3c-e503-4bee-8254-94317035f5ce-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:25:09.393116 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.393083    2558 generic.go:356] "Generic (PLEG): container finished" podID="45321d3c-e503-4bee-8254-94317035f5ce" containerID="0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3" exitCode=0
Jul 10 19:25:09.393505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.393134    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"
Jul 10 19:25:09.393505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.393158    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerDied","Data":"0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3"}
Jul 10 19:25:09.393505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.393220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl" event={"ID":"45321d3c-e503-4bee-8254-94317035f5ce","Type":"ContainerDied","Data":"05dcd2dd81fd03f81907ae5acb2ca06965a269d5d4948598bbf06df5997366de"}
Jul 10 19:25:09.393505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.393249    2558 scope.go:117] "RemoveContainer" containerID="97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785"
Jul 10 19:25:09.393725 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.393683    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:25:09.401177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.401154    2558 scope.go:117] "RemoveContainer" containerID="0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3"
Jul 10 19:25:09.408174 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.408101    2558 scope.go:117] "RemoveContainer" containerID="e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37"
Jul 10 19:25:09.410375 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.410354    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"]
Jul 10 19:25:09.413708 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.413690    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-8cb957f44-h2vdl"]
Jul 10 19:25:09.416263 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.416246    2558 scope.go:117] "RemoveContainer" containerID="97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785"
Jul 10 19:25:09.416490 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:25:09.416475    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785\": container with ID starting with 97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785 not found: ID does not exist" containerID="97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785"
Jul 10 19:25:09.416538 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.416500    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785"} err="failed to get container status \"97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785\": rpc error: code = NotFound desc = could not find container \"97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785\": container with ID starting with 97aefcb95e2c6ad8ab081ce0b4046b164e02c1871c37ff928d14e6bf5a58b785 not found: ID does not exist"
Jul 10 19:25:09.416538 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.416522    2558 scope.go:117] "RemoveContainer" containerID="0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3"
Jul 10 19:25:09.416835 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:25:09.416764    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3\": container with ID starting with 0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3 not found: ID does not exist" containerID="0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3"
Jul 10 19:25:09.416835 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.416796    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3"} err="failed to get container status \"0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3\": rpc error: code = NotFound desc = could not find container \"0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3\": container with ID starting with 0dd60aa2a730c5bee668c3c43e115219992dc9cefd6a3d6a74d8c958ef31b9d3 not found: ID does not exist"
Jul 10 19:25:09.416835 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.416818    2558 scope.go:117] "RemoveContainer" containerID="e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37"
Jul 10 19:25:09.417053 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:25:09.417036    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37\": container with ID starting with e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37 not found: ID does not exist" containerID="e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37"
Jul 10 19:25:09.417096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:09.417057    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37"} err="failed to get container status \"e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37\": rpc error: code = NotFound desc = could not find container \"e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37\": container with ID starting with e9e5b634f7adc131754277b2af1e7de13a442ad901ad83ba297750fbebc47b37 not found: ID does not exist"
Jul 10 19:25:11.257102 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:11.257067    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45321d3c-e503-4bee-8254-94317035f5ce" path="/var/lib/kubelet/pods/45321d3c-e503-4bee-8254-94317035f5ce/volumes"
Jul 10 19:25:14.398649 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:14.398620    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:25:14.399344 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:14.399316    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:25:24.400046 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:24.400010    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:25:34.400028 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:34.399980    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:25:44.399314 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:44.399225    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:25:54.399959 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:25:54.399920    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:26:04.399597 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:04.399557    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:26:14.400384 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:14.400343    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:26:21.688958 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.688921    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"]
Jul 10 19:26:21.689352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.689195    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" containerID="cri-o://1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101" gracePeriod=30
Jul 10 19:26:21.689352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.689230    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kube-rbac-proxy" containerID="cri-o://c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b" gracePeriod=30
Jul 10 19:26:21.786172 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786137    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"]
Jul 10 19:26:21.786465 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786448    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kube-rbac-proxy"
Jul 10 19:26:21.786575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786468    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kube-rbac-proxy"
Jul 10 19:26:21.786575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786485    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kserve-container"
Jul 10 19:26:21.786575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786493    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kserve-container"
Jul 10 19:26:21.786575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786505    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="storage-initializer"
Jul 10 19:26:21.786575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786514    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="storage-initializer"
Jul 10 19:26:21.786575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786575    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kserve-container"
Jul 10 19:26:21.786932 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.786586    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45321d3c-e503-4bee-8254-94317035f5ce" containerName="kube-rbac-proxy"
Jul 10 19:26:21.789689 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.789667    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.792826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.792805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 10 19:26:21.792932 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.792811    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 10 19:26:21.802134 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.802114    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"]
Jul 10 19:26:21.807121 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.807095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6q6k\" (UniqueName: \"kubernetes.io/projected/310b2986-d62b-49d3-aa32-8938e0a877d7-kube-api-access-b6q6k\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.807300 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.807273    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310b2986-d62b-49d3-aa32-8938e0a877d7-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.807424 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.807311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310b2986-d62b-49d3-aa32-8938e0a877d7-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.807424 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.807339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310b2986-d62b-49d3-aa32-8938e0a877d7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.907848 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.907817    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310b2986-d62b-49d3-aa32-8938e0a877d7-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.907848 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.907852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310b2986-d62b-49d3-aa32-8938e0a877d7-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.908062 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.907874    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310b2986-d62b-49d3-aa32-8938e0a877d7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.908062 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.907951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6q6k\" (UniqueName: \"kubernetes.io/projected/310b2986-d62b-49d3-aa32-8938e0a877d7-kube-api-access-b6q6k\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.908233 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.908215    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310b2986-d62b-49d3-aa32-8938e0a877d7-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.908656 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.908631    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310b2986-d62b-49d3-aa32-8938e0a877d7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.910399 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.910372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310b2986-d62b-49d3-aa32-8938e0a877d7-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:21.917211 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:21.917191    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6q6k\" (UniqueName: \"kubernetes.io/projected/310b2986-d62b-49d3-aa32-8938e0a877d7-kube-api-access-b6q6k\") pod \"isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:22.099307 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:22.099278    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:22.235758 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:22.235710    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"]
Jul 10 19:26:22.238217 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:26:22.238172    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310b2986_d62b_49d3_aa32_8938e0a877d7.slice/crio-3c3b1e2ae001faeb0c3dabc9cae425c37cf3e68dff7789a755ec709c0280a9ea WatchSource:0}: Error finding container 3c3b1e2ae001faeb0c3dabc9cae425c37cf3e68dff7789a755ec709c0280a9ea: Status 404 returned error can't find the container with id 3c3b1e2ae001faeb0c3dabc9cae425c37cf3e68dff7789a755ec709c0280a9ea
Jul 10 19:26:22.645936 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:22.645902    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerStarted","Data":"82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a"}
Jul 10 19:26:22.646143 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:22.645971    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerStarted","Data":"3c3b1e2ae001faeb0c3dabc9cae425c37cf3e68dff7789a755ec709c0280a9ea"}
Jul 10 19:26:22.648099 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:22.648073    2558 generic.go:356] "Generic (PLEG): container finished" podID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerID="c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b" exitCode=2
Jul 10 19:26:22.648206 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:22.648146    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerDied","Data":"c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b"}
Jul 10 19:26:24.394828 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:24.394785    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Jul 10 19:26:24.400182 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:24.400155    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Jul 10 19:26:25.835515 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.835488    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:26:25.930614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.930547    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls\") pod \"4ba5c739-279a-4a09-9686-2acac54f14a1\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") "
Jul 10 19:26:25.930614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.930587    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5c739-279a-4a09-9686-2acac54f14a1-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"4ba5c739-279a-4a09-9686-2acac54f14a1\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") "
Jul 10 19:26:25.930806 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.930627    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xjv9\" (UniqueName: \"kubernetes.io/projected/4ba5c739-279a-4a09-9686-2acac54f14a1-kube-api-access-9xjv9\") pod \"4ba5c739-279a-4a09-9686-2acac54f14a1\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") "
Jul 10 19:26:25.930806 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.930670    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5c739-279a-4a09-9686-2acac54f14a1-kserve-provision-location\") pod \"4ba5c739-279a-4a09-9686-2acac54f14a1\" (UID: \"4ba5c739-279a-4a09-9686-2acac54f14a1\") "
Jul 10 19:26:25.931018 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.930995    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba5c739-279a-4a09-9686-2acac54f14a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ba5c739-279a-4a09-9686-2acac54f14a1" (UID: "4ba5c739-279a-4a09-9686-2acac54f14a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:26:25.931062 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.930998    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba5c739-279a-4a09-9686-2acac54f14a1-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "4ba5c739-279a-4a09-9686-2acac54f14a1" (UID: "4ba5c739-279a-4a09-9686-2acac54f14a1"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:26:25.932645 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.932617    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba5c739-279a-4a09-9686-2acac54f14a1-kube-api-access-9xjv9" (OuterVolumeSpecName: "kube-api-access-9xjv9") pod "4ba5c739-279a-4a09-9686-2acac54f14a1" (UID: "4ba5c739-279a-4a09-9686-2acac54f14a1"). InnerVolumeSpecName "kube-api-access-9xjv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:26:25.932773 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:25.932704    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ba5c739-279a-4a09-9686-2acac54f14a1" (UID: "4ba5c739-279a-4a09-9686-2acac54f14a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:26:26.031288 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.031260    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjv9\" (UniqueName: \"kubernetes.io/projected/4ba5c739-279a-4a09-9686-2acac54f14a1-kube-api-access-9xjv9\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:26:26.031288 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.031282    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba5c739-279a-4a09-9686-2acac54f14a1-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:26:26.031288 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.031292    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba5c739-279a-4a09-9686-2acac54f14a1-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:26:26.031479 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.031301    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba5c739-279a-4a09-9686-2acac54f14a1-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:26:26.665071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.665040    2558 generic.go:356] "Generic (PLEG): container finished" podID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerID="82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a" exitCode=0
Jul 10 19:26:26.665272 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.665120    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerDied","Data":"82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a"}
Jul 10 19:26:26.666977 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.666955    2558 generic.go:356] "Generic (PLEG): container finished" podID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerID="1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101" exitCode=0
Jul 10 19:26:26.667059 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.667005    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"
Jul 10 19:26:26.667106 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.667005    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerDied","Data":"1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101"}
Jul 10 19:26:26.667142 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.667129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j" event={"ID":"4ba5c739-279a-4a09-9686-2acac54f14a1","Type":"ContainerDied","Data":"277f72011b9c0a1ec0ef331e803b2afda7d6f3ba4a7c6bf30a000eb9c80ced28"}
Jul 10 19:26:26.667176 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.667161    2558 scope.go:117] "RemoveContainer" containerID="c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b"
Jul 10 19:26:26.678981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.678964    2558 scope.go:117] "RemoveContainer" containerID="1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101"
Jul 10 19:26:26.690462 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.690447    2558 scope.go:117] "RemoveContainer" containerID="a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e"
Jul 10 19:26:26.702439 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.702420    2558 scope.go:117] "RemoveContainer" containerID="c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b"
Jul 10 19:26:26.702790 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:26:26.702768    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b\": container with ID starting with c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b not found: ID does not exist" containerID="c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b"
Jul 10 19:26:26.702864 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.702799    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b"} err="failed to get container status \"c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b\": rpc error: code = NotFound desc = could not find container \"c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b\": container with ID starting with c93f607378f918b580862ea0df43721b2fa4e76e00b6319e6d763211501ccc2b not found: ID does not exist"
Jul 10 19:26:26.702864 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.702817    2558 scope.go:117] "RemoveContainer" containerID="1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101"
Jul 10 19:26:26.703055 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:26:26.703039    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101\": container with ID starting with 1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101 not found: ID does not exist" containerID="1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101"
Jul 10 19:26:26.703111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.703060    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101"} err="failed to get container status \"1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101\": rpc error: code = NotFound desc = could not find container \"1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101\": container with ID starting with 1f4bb0ded40c5f00069ca012dc69319e4a9c1a99b820854833a1f33edfd66101 not found: ID does not exist"
Jul 10 19:26:26.703111 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.703075    2558 scope.go:117] "RemoveContainer" containerID="a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e"
Jul 10 19:26:26.703324 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:26:26.703299    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e\": container with ID starting with a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e not found: ID does not exist" containerID="a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e"
Jul 10 19:26:26.703415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.703332    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e"} err="failed to get container status \"a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e\": rpc error: code = NotFound desc = could not find container \"a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e\": container with ID starting with a25041a376a94137d9b107dc468dad79aadb708d6a58bacf3635f6bd64eb079e not found: ID does not exist"
Jul 10 19:26:26.727296 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.727276    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"]
Jul 10 19:26:26.732206 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:26.732184    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-66758b956b-5nz8j"]
Jul 10 19:26:27.257903 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:27.257873    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" path="/var/lib/kubelet/pods/4ba5c739-279a-4a09-9686-2acac54f14a1/volumes"
Jul 10 19:26:27.673964 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:27.673883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerStarted","Data":"fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c"}
Jul 10 19:26:27.673964 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:27.673941    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerStarted","Data":"b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739"}
Jul 10 19:26:27.674262 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:27.674245    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:27.694380 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:27.694335    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podStartSLOduration=3.146955488 podStartE2EDuration="6.69432173s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.547366242s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:26:27.692522299 +0000 UTC m=+2832.954187073" watchObservedRunningTime="2026-07-10 19:26:27.69432173 +0000 UTC m=+2832.955986503"
Jul 10 19:26:28.678609 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:28.678578    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:28.679869 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:28.679843    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:26:29.681730 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:29.681688    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:26:34.686713 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:34.686686    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:26:34.687282 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:34.687256    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:26:44.687625 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:44.687585    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:26:54.687519 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:26:54.687478    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:27:04.687441 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:04.687401    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:27:14.687325 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:14.687288    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:27:24.688097 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:24.688057    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:27:34.687903 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:34.687870    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:27:41.829626 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.829594    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"]
Jul 10 19:27:41.830094 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.829875    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" containerID="cri-o://b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739" gracePeriod=30
Jul 10 19:27:41.830094 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.829980    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kube-rbac-proxy" containerID="cri-o://fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c" gracePeriod=30
Jul 10 19:27:41.925883 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.925849    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"]
Jul 10 19:27:41.926137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926125    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kube-rbac-proxy"
Jul 10 19:27:41.926187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926138    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kube-rbac-proxy"
Jul 10 19:27:41.926187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926149    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="storage-initializer"
Jul 10 19:27:41.926187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926155    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="storage-initializer"
Jul 10 19:27:41.926187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926163    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container"
Jul 10 19:27:41.926187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926168    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container"
Jul 10 19:27:41.926397 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926214    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kube-rbac-proxy"
Jul 10 19:27:41.926397 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.926223    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba5c739-279a-4a09-9686-2acac54f14a1" containerName="kserve-container"
Jul 10 19:27:41.929158 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.929143    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:41.931610 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.931589    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 10 19:27:41.932656 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.932632    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 10 19:27:41.939687 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:41.939661    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"]
Jul 10 19:27:42.071612 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.071580    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e223fa14-eb03-4997-a6cd-131e0e8b49c6-proxy-tls\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.071825 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.071624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hhn\" (UniqueName: \"kubernetes.io/projected/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kube-api-access-d5hhn\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.071825 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.071698    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kserve-provision-location\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.071825 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.071772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e223fa14-eb03-4997-a6cd-131e0e8b49c6-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.172963 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.172883    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kserve-provision-location\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.172963 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.172931    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e223fa14-eb03-4997-a6cd-131e0e8b49c6-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.173174 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.172975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e223fa14-eb03-4997-a6cd-131e0e8b49c6-proxy-tls\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.173174 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.173109    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hhn\" (UniqueName: \"kubernetes.io/projected/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kube-api-access-d5hhn\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.173261 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.173203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kserve-provision-location\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.173648 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.173624    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e223fa14-eb03-4997-a6cd-131e0e8b49c6-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.175319 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.175293    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e223fa14-eb03-4997-a6cd-131e0e8b49c6-proxy-tls\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.181252 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.181235    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hhn\" (UniqueName: \"kubernetes.io/projected/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kube-api-access-d5hhn\") pod \"isvc-tensorflow-predictor-7b797b7849-whcs9\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.242151 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.242121    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:42.368669 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.368647    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"]
Jul 10 19:27:42.371065 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:27:42.371039    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode223fa14_eb03_4997_a6cd_131e0e8b49c6.slice/crio-8759d9ff8427df089cad78073735018b88a0f30e9480952ed9699c5a91b46f55 WatchSource:0}: Error finding container 8759d9ff8427df089cad78073735018b88a0f30e9480952ed9699c5a91b46f55: Status 404 returned error can't find the container with id 8759d9ff8427df089cad78073735018b88a0f30e9480952ed9699c5a91b46f55
Jul 10 19:27:42.937192 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.937155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerStarted","Data":"6cac461014684c8a461b224e1d953ac454ba8cc87576056da71cad1a6ebb1794"}
Jul 10 19:27:42.937611 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.937220    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerStarted","Data":"8759d9ff8427df089cad78073735018b88a0f30e9480952ed9699c5a91b46f55"}
Jul 10 19:27:42.939065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.939036    2558 generic.go:356] "Generic (PLEG): container finished" podID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerID="fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c" exitCode=2
Jul 10 19:27:42.939191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:42.939070    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerDied","Data":"fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c"}
Jul 10 19:27:44.682882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:44.682843    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Jul 10 19:27:44.687847 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:44.687820    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Jul 10 19:27:45.773043 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.773019    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:27:45.902327 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.902252    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310b2986-d62b-49d3-aa32-8938e0a877d7-proxy-tls\") pod \"310b2986-d62b-49d3-aa32-8938e0a877d7\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") "
Jul 10 19:27:45.902327 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.902288    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310b2986-d62b-49d3-aa32-8938e0a877d7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"310b2986-d62b-49d3-aa32-8938e0a877d7\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") "
Jul 10 19:27:45.902555 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.902336    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310b2986-d62b-49d3-aa32-8938e0a877d7-kserve-provision-location\") pod \"310b2986-d62b-49d3-aa32-8938e0a877d7\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") "
Jul 10 19:27:45.902555 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.902372    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6q6k\" (UniqueName: \"kubernetes.io/projected/310b2986-d62b-49d3-aa32-8938e0a877d7-kube-api-access-b6q6k\") pod \"310b2986-d62b-49d3-aa32-8938e0a877d7\" (UID: \"310b2986-d62b-49d3-aa32-8938e0a877d7\") "
Jul 10 19:27:45.902687 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.902663    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310b2986-d62b-49d3-aa32-8938e0a877d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "310b2986-d62b-49d3-aa32-8938e0a877d7" (UID: "310b2986-d62b-49d3-aa32-8938e0a877d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:27:45.902788 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.902766    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310b2986-d62b-49d3-aa32-8938e0a877d7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "310b2986-d62b-49d3-aa32-8938e0a877d7" (UID: "310b2986-d62b-49d3-aa32-8938e0a877d7"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:27:45.904287 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.904266    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310b2986-d62b-49d3-aa32-8938e0a877d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "310b2986-d62b-49d3-aa32-8938e0a877d7" (UID: "310b2986-d62b-49d3-aa32-8938e0a877d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:27:45.904372 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.904354    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310b2986-d62b-49d3-aa32-8938e0a877d7-kube-api-access-b6q6k" (OuterVolumeSpecName: "kube-api-access-b6q6k") pod "310b2986-d62b-49d3-aa32-8938e0a877d7" (UID: "310b2986-d62b-49d3-aa32-8938e0a877d7"). InnerVolumeSpecName "kube-api-access-b6q6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:27:45.950698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.950670    2558 generic.go:356] "Generic (PLEG): container finished" podID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerID="b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739" exitCode=0
Jul 10 19:27:45.950871 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.950710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerDied","Data":"b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739"}
Jul 10 19:27:45.950871 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.950733    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"
Jul 10 19:27:45.950871 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.950776    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw" event={"ID":"310b2986-d62b-49d3-aa32-8938e0a877d7","Type":"ContainerDied","Data":"3c3b1e2ae001faeb0c3dabc9cae425c37cf3e68dff7789a755ec709c0280a9ea"}
Jul 10 19:27:45.950871 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.950801    2558 scope.go:117] "RemoveContainer" containerID="fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c"
Jul 10 19:27:45.958995 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.958979    2558 scope.go:117] "RemoveContainer" containerID="b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739"
Jul 10 19:27:45.965627 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.965609    2558 scope.go:117] "RemoveContainer" containerID="82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a"
Jul 10 19:27:45.972328 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.972310    2558 scope.go:117] "RemoveContainer" containerID="fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c"
Jul 10 19:27:45.972559 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:27:45.972541    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c\": container with ID starting with fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c not found: ID does not exist" containerID="fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c"
Jul 10 19:27:45.972611 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.972567    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c"} err="failed to get container status \"fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c\": rpc error: code = NotFound desc = could not find container \"fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c\": container with ID starting with fb183bfb8aebd1bd52c680afada4e2dce894758b4b2a741f62df058e48559b1c not found: ID does not exist"
Jul 10 19:27:45.972611 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.972584    2558 scope.go:117] "RemoveContainer" containerID="b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739"
Jul 10 19:27:45.972833 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:27:45.972815    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739\": container with ID starting with b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739 not found: ID does not exist" containerID="b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739"
Jul 10 19:27:45.972892 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.972839    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739"} err="failed to get container status \"b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739\": rpc error: code = NotFound desc = could not find container \"b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739\": container with ID starting with b914d01989ed138704d7020a0a89c55a2685f61e25e12531b17e80f379fcd739 not found: ID does not exist"
Jul 10 19:27:45.972892 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.972854    2558 scope.go:117] "RemoveContainer" containerID="82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a"
Jul 10 19:27:45.973055 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:27:45.973037    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a\": container with ID starting with 82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a not found: ID does not exist" containerID="82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a"
Jul 10 19:27:45.973095 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.973059    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a"} err="failed to get container status \"82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a\": rpc error: code = NotFound desc = could not find container \"82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a\": container with ID starting with 82c5d58ec6b02bbae36d1279c5b7765b6521043e56a1d44b0721a384ac9dc64a not found: ID does not exist"
Jul 10 19:27:45.982820 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.982800    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"]
Jul 10 19:27:45.988406 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:45.988386    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-857b7c4bdc-c48gw"]
Jul 10 19:27:46.003872 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:46.003850    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6q6k\" (UniqueName: \"kubernetes.io/projected/310b2986-d62b-49d3-aa32-8938e0a877d7-kube-api-access-b6q6k\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:27:46.003872 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:46.003869    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310b2986-d62b-49d3-aa32-8938e0a877d7-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:27:46.003998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:46.003881    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310b2986-d62b-49d3-aa32-8938e0a877d7-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:27:46.003998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:46.003892    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310b2986-d62b-49d3-aa32-8938e0a877d7-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:27:46.955954 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:46.955920    2558 generic.go:356] "Generic (PLEG): container finished" podID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerID="6cac461014684c8a461b224e1d953ac454ba8cc87576056da71cad1a6ebb1794" exitCode=0
Jul 10 19:27:46.956369 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:46.955993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerDied","Data":"6cac461014684c8a461b224e1d953ac454ba8cc87576056da71cad1a6ebb1794"}
Jul 10 19:27:46.957186 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:46.957170    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:27:47.256603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:47.256571    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" path="/var/lib/kubelet/pods/310b2986-d62b-49d3-aa32-8938e0a877d7/volumes"
Jul 10 19:27:51.976177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:51.976134    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerStarted","Data":"030fbd2311aa3564a185503bd1d245bc140fe3d9236c8fbf785532240d862e94"}
Jul 10 19:27:51.976643 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:51.976205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerStarted","Data":"1c816e82b645a854c3c0a4925005d0068215ba42151afd408f505828fe229d10"}
Jul 10 19:27:51.976643 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:51.976422    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:51.996877 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:51.996814    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podStartSLOduration=2.545778587 podStartE2EDuration="10.996799839s" totalImagesPullingTime="4.060286905s" totalInitContainerRuntime="4.390734347s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:27:51.99549868 +0000 UTC m=+2917.257163476" watchObservedRunningTime="2026-07-10 19:27:51.996799839 +0000 UTC m=+2917.258464669"
Jul 10 19:27:52.979801 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:52.979767    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:52.980951 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:52.980924    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 10 19:27:53.987940 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:53.987903    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 10 19:27:58.992431 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:58.992404    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:27:58.992901 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:27:58.992804    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Jul 10 19:28:08.993074 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:08.993043    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:28:21.954328 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:21.954282    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"]
Jul 10 19:28:21.954756 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:21.954610    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kserve-container" containerID="cri-o://1c816e82b645a854c3c0a4925005d0068215ba42151afd408f505828fe229d10" gracePeriod=30
Jul 10 19:28:21.954756 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:21.954674    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy" containerID="cri-o://030fbd2311aa3564a185503bd1d245bc140fe3d9236c8fbf785532240d862e94" gracePeriod=30
Jul 10 19:28:22.020858 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.020828    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"]
Jul 10 19:28:22.021086 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021074    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container"
Jul 10 19:28:22.021131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021088    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container"
Jul 10 19:28:22.021131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021106    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="storage-initializer"
Jul 10 19:28:22.021131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021111    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="storage-initializer"
Jul 10 19:28:22.021131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021117    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kube-rbac-proxy"
Jul 10 19:28:22.021131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021123    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kube-rbac-proxy"
Jul 10 19:28:22.021287 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021166    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kserve-container"
Jul 10 19:28:22.021287 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.021175    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="310b2986-d62b-49d3-aa32-8938e0a877d7" containerName="kube-rbac-proxy"
Jul 10 19:28:22.027290 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.027270    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.029450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.029433    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 10 19:28:22.029588 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.029462    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:28:22.034667 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.034604    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"]
Jul 10 19:28:22.052496 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.052470    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435fbc3c-a34f-4bdd-a04b-167d64b6c349-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.052591 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.052509    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435fbc3c-a34f-4bdd-a04b-167d64b6c349-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.052591 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.052543    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbwm\" (UniqueName: \"kubernetes.io/projected/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kube-api-access-rxbwm\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.052684 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.052617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.085304 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.085282    2558 generic.go:356] "Generic (PLEG): container finished" podID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerID="030fbd2311aa3564a185503bd1d245bc140fe3d9236c8fbf785532240d862e94" exitCode=2
Jul 10 19:28:22.085398 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.085350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerDied","Data":"030fbd2311aa3564a185503bd1d245bc140fe3d9236c8fbf785532240d862e94"}
Jul 10 19:28:22.152970 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.152946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.153072 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.152979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435fbc3c-a34f-4bdd-a04b-167d64b6c349-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.153072 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.153014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435fbc3c-a34f-4bdd-a04b-167d64b6c349-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.153072 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.153062    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbwm\" (UniqueName: \"kubernetes.io/projected/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kube-api-access-rxbwm\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.153338 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.153322    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.153697 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.153681    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435fbc3c-a34f-4bdd-a04b-167d64b6c349-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.155339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.155320    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435fbc3c-a34f-4bdd-a04b-167d64b6c349-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.160660 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.160640    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbwm\" (UniqueName: \"kubernetes.io/projected/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kube-api-access-rxbwm\") pod \"isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.338336 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.338303    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:22.466164 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:22.466134    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"]
Jul 10 19:28:22.469082 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:28:22.469055    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435fbc3c_a34f_4bdd_a04b_167d64b6c349.slice/crio-e4b619d767fef70a1d657197b3dd40881535eb7041d55b33e17986c9f43be5ce WatchSource:0}: Error finding container e4b619d767fef70a1d657197b3dd40881535eb7041d55b33e17986c9f43be5ce: Status 404 returned error can't find the container with id e4b619d767fef70a1d657197b3dd40881535eb7041d55b33e17986c9f43be5ce
Jul 10 19:28:23.089914 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:23.089875    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerStarted","Data":"99200cfa4b2069fe67fd3a1d6b338670976894daf10e0bf0f1fe9cd7405aa6f2"}
Jul 10 19:28:23.090312 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:23.089953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerStarted","Data":"e4b619d767fef70a1d657197b3dd40881535eb7041d55b33e17986c9f43be5ce"}
Jul 10 19:28:23.988546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:23.988503    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 10 19:28:27.106529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:27.106447    2558 generic.go:356] "Generic (PLEG): container finished" podID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerID="99200cfa4b2069fe67fd3a1d6b338670976894daf10e0bf0f1fe9cd7405aa6f2" exitCode=0
Jul 10 19:28:27.106529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:27.106492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerDied","Data":"99200cfa4b2069fe67fd3a1d6b338670976894daf10e0bf0f1fe9cd7405aa6f2"}
Jul 10 19:28:28.112196 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:28.112164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerStarted","Data":"e6c70b83e084aa8e2ea9aa6b44b18867b9dd81577d90c69a602b1634144702e7"}
Jul 10 19:28:28.112574 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:28.112226    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerStarted","Data":"ea0cfb87b169667f6cbb9510fd5eb5be09fc04fd17f170b3734e6c486ab074a0"}
Jul 10 19:28:28.112574 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:28.112423    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:28.132415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:28.132363    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podStartSLOduration=1.821195655 podStartE2EDuration="6.132350246s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.311154591s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:28:28.131013086 +0000 UTC m=+2953.392677894" watchObservedRunningTime="2026-07-10 19:28:28.132350246 +0000 UTC m=+2953.394015083"
Jul 10 19:28:28.988331 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:28.988290    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 10 19:28:29.116164 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:29.116131    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:29.117156 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:29.117128    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 10 19:28:30.119469 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:30.119429    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 10 19:28:33.988702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:33.988656    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 10 19:28:33.989103 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:33.988816    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:28:35.123972 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:35.123945    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:35.124436 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:35.124413    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Jul 10 19:28:38.988694 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:38.988609    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 10 19:28:43.987997 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:43.987953    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 10 19:28:45.125166 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:45.125139    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:28:48.988518 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:48.988471    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Jul 10 19:28:52.201562 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.201531    2558 generic.go:356] "Generic (PLEG): container finished" podID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerID="1c816e82b645a854c3c0a4925005d0068215ba42151afd408f505828fe229d10" exitCode=137
Jul 10 19:28:52.201932 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.201569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerDied","Data":"1c816e82b645a854c3c0a4925005d0068215ba42151afd408f505828fe229d10"}
Jul 10 19:28:52.604345 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.604323    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:28:52.671085 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.671055    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e223fa14-eb03-4997-a6cd-131e0e8b49c6-isvc-tensorflow-kube-rbac-proxy-sar-config\") pod \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") "
Jul 10 19:28:52.671252 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.671095    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e223fa14-eb03-4997-a6cd-131e0e8b49c6-proxy-tls\") pod \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") "
Jul 10 19:28:52.671252 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.671133    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kserve-provision-location\") pod \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") "
Jul 10 19:28:52.671380 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.671299    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5hhn\" (UniqueName: \"kubernetes.io/projected/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kube-api-access-d5hhn\") pod \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\" (UID: \"e223fa14-eb03-4997-a6cd-131e0e8b49c6\") "
Jul 10 19:28:52.671488 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.671467    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e223fa14-eb03-4997-a6cd-131e0e8b49c6-isvc-tensorflow-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-kube-rbac-proxy-sar-config") pod "e223fa14-eb03-4997-a6cd-131e0e8b49c6" (UID: "e223fa14-eb03-4997-a6cd-131e0e8b49c6"). InnerVolumeSpecName "isvc-tensorflow-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:28:52.673189 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.673159    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kube-api-access-d5hhn" (OuterVolumeSpecName: "kube-api-access-d5hhn") pod "e223fa14-eb03-4997-a6cd-131e0e8b49c6" (UID: "e223fa14-eb03-4997-a6cd-131e0e8b49c6"). InnerVolumeSpecName "kube-api-access-d5hhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:28:52.673287 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.673200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e223fa14-eb03-4997-a6cd-131e0e8b49c6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e223fa14-eb03-4997-a6cd-131e0e8b49c6" (UID: "e223fa14-eb03-4997-a6cd-131e0e8b49c6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:28:52.687145 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.687122    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e223fa14-eb03-4997-a6cd-131e0e8b49c6" (UID: "e223fa14-eb03-4997-a6cd-131e0e8b49c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:28:52.772063 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.772037    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5hhn\" (UniqueName: \"kubernetes.io/projected/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kube-api-access-d5hhn\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:28:52.772063 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.772065    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e223fa14-eb03-4997-a6cd-131e0e8b49c6-isvc-tensorflow-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:28:52.772222 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.772075    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e223fa14-eb03-4997-a6cd-131e0e8b49c6-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:28:52.772222 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:52.772085    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e223fa14-eb03-4997-a6cd-131e0e8b49c6-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:28:53.206624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.206537    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9" event={"ID":"e223fa14-eb03-4997-a6cd-131e0e8b49c6","Type":"ContainerDied","Data":"8759d9ff8427df089cad78073735018b88a0f30e9480952ed9699c5a91b46f55"}
Jul 10 19:28:53.206624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.206598    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"
Jul 10 19:28:53.207098 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.206603    2558 scope.go:117] "RemoveContainer" containerID="030fbd2311aa3564a185503bd1d245bc140fe3d9236c8fbf785532240d862e94"
Jul 10 19:28:53.215721 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.215706    2558 scope.go:117] "RemoveContainer" containerID="1c816e82b645a854c3c0a4925005d0068215ba42151afd408f505828fe229d10"
Jul 10 19:28:53.222837 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.222816    2558 scope.go:117] "RemoveContainer" containerID="6cac461014684c8a461b224e1d953ac454ba8cc87576056da71cad1a6ebb1794"
Jul 10 19:28:53.230985 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.230938    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"]
Jul 10 19:28:53.236209 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.236186    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-7b797b7849-whcs9"]
Jul 10 19:28:53.256314 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:28:53.256289    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" path="/var/lib/kubelet/pods/e223fa14-eb03-4997-a6cd-131e0e8b49c6/volumes"
Jul 10 19:29:02.140036 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.140001    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"]
Jul 10 19:29:02.140624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.140294    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kserve-container" containerID="cri-o://ea0cfb87b169667f6cbb9510fd5eb5be09fc04fd17f170b3734e6c486ab074a0" gracePeriod=30
Jul 10 19:29:02.140624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.140304    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy" containerID="cri-o://e6c70b83e084aa8e2ea9aa6b44b18867b9dd81577d90c69a602b1634144702e7" gracePeriod=30
Jul 10 19:29:02.222227 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222192    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"]
Jul 10 19:29:02.222483 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222469    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="storage-initializer"
Jul 10 19:29:02.222483 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222484    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="storage-initializer"
Jul 10 19:29:02.222613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222493    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kserve-container"
Jul 10 19:29:02.222613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222499    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kserve-container"
Jul 10 19:29:02.222613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222516    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy"
Jul 10 19:29:02.222613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222522    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy"
Jul 10 19:29:02.222613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222560    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kube-rbac-proxy"
Jul 10 19:29:02.222613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.222570    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e223fa14-eb03-4997-a6cd-131e0e8b49c6" containerName="kserve-container"
Jul 10 19:29:02.225580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.225559    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.227977 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.227954    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 10 19:29:02.228091 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.228072    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 10 19:29:02.235838 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.235806    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"]
Jul 10 19:29:02.341137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.341108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.341137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.341139    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-proxy-tls\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.341395 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.341241    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlz4\" (UniqueName: \"kubernetes.io/projected/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kube-api-access-5tlz4\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.341395 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.341328    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kserve-provision-location\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.441700 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.441627    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.441700 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.441657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-proxy-tls\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.441700 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.441687    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlz4\" (UniqueName: \"kubernetes.io/projected/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kube-api-access-5tlz4\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.442015 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.441714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kserve-provision-location\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.442143 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.442124    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kserve-provision-location\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.442325 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.442306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-isvc-triton-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.444241 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.444219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-proxy-tls\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.450221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.450199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlz4\" (UniqueName: \"kubernetes.io/projected/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kube-api-access-5tlz4\") pod \"isvc-triton-predictor-6cfd998464-2ncxb\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.535721 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.535689    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:29:02.661058 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:02.661006    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"]
Jul 10 19:29:02.663235 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:29:02.663212    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6dfcf8_3934_4a28_ac84_b4b8126bfc2f.slice/crio-fead6b248ec4e2ef54f1e971239d8352caf09a06c1650e7d9e7869f8d270abc2 WatchSource:0}: Error finding container fead6b248ec4e2ef54f1e971239d8352caf09a06c1650e7d9e7869f8d270abc2: Status 404 returned error can't find the container with id fead6b248ec4e2ef54f1e971239d8352caf09a06c1650e7d9e7869f8d270abc2
Jul 10 19:29:03.243343 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:03.243310    2558 generic.go:356] "Generic (PLEG): container finished" podID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerID="e6c70b83e084aa8e2ea9aa6b44b18867b9dd81577d90c69a602b1634144702e7" exitCode=2
Jul 10 19:29:03.243719 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:03.243384    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerDied","Data":"e6c70b83e084aa8e2ea9aa6b44b18867b9dd81577d90c69a602b1634144702e7"}
Jul 10 19:29:03.244703 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:03.244681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerStarted","Data":"09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42"}
Jul 10 19:29:03.244850 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:03.244733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerStarted","Data":"fead6b248ec4e2ef54f1e971239d8352caf09a06c1650e7d9e7869f8d270abc2"}
Jul 10 19:29:05.120555 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:05.120515    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 10 19:29:07.260512 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:07.260480    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerID="09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42" exitCode=0
Jul 10 19:29:07.260910 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:07.260525    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerDied","Data":"09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42"}
Jul 10 19:29:10.121284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:10.120869    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 10 19:29:15.120686 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:15.120640    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 10 19:29:15.121513 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:15.121270    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:29:15.414323 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:15.414238    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:29:18.652279 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:18.652247    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:29:20.120212 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:20.120159    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 10 19:29:25.119881 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:25.119837    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 10 19:29:30.119698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:30.119656    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Jul 10 19:29:32.404362 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:32.404263    2558 generic.go:356] "Generic (PLEG): container finished" podID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerID="ea0cfb87b169667f6cbb9510fd5eb5be09fc04fd17f170b3734e6c486ab074a0" exitCode=137
Jul 10 19:29:32.404362 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:32.404311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerDied","Data":"ea0cfb87b169667f6cbb9510fd5eb5be09fc04fd17f170b3734e6c486ab074a0"}
Jul 10 19:29:32.856198 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:32.856129    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:29:33.000187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.000144    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435fbc3c-a34f-4bdd-a04b-167d64b6c349-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") pod \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") "
Jul 10 19:29:33.000426 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.000194    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxbwm\" (UniqueName: \"kubernetes.io/projected/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kube-api-access-rxbwm\") pod \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") "
Jul 10 19:29:33.000426 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.000274    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435fbc3c-a34f-4bdd-a04b-167d64b6c349-proxy-tls\") pod \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") "
Jul 10 19:29:33.000426 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.000305    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kserve-provision-location\") pod \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\" (UID: \"435fbc3c-a34f-4bdd-a04b-167d64b6c349\") "
Jul 10 19:29:33.000665 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.000522    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435fbc3c-a34f-4bdd-a04b-167d64b6c349-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config") pod "435fbc3c-a34f-4bdd-a04b-167d64b6c349" (UID: "435fbc3c-a34f-4bdd-a04b-167d64b6c349"). InnerVolumeSpecName "isvc-tensorflow-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:29:33.003404 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.003365    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/435fbc3c-a34f-4bdd-a04b-167d64b6c349-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "435fbc3c-a34f-4bdd-a04b-167d64b6c349" (UID: "435fbc3c-a34f-4bdd-a04b-167d64b6c349"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:29:33.003590 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.003561    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kube-api-access-rxbwm" (OuterVolumeSpecName: "kube-api-access-rxbwm") pod "435fbc3c-a34f-4bdd-a04b-167d64b6c349" (UID: "435fbc3c-a34f-4bdd-a04b-167d64b6c349"). InnerVolumeSpecName "kube-api-access-rxbwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:29:33.005096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.005071    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "435fbc3c-a34f-4bdd-a04b-167d64b6c349" (UID: "435fbc3c-a34f-4bdd-a04b-167d64b6c349"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:29:33.101115 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.101082    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbwm\" (UniqueName: \"kubernetes.io/projected/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kube-api-access-rxbwm\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:29:33.101115 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.101114    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/435fbc3c-a34f-4bdd-a04b-167d64b6c349-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:29:33.101341 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.101129    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/435fbc3c-a34f-4bdd-a04b-167d64b6c349-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:29:33.101341 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.101144    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/435fbc3c-a34f-4bdd-a04b-167d64b6c349-isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:29:33.411278 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.411249    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"
Jul 10 19:29:33.411819 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.411248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn" event={"ID":"435fbc3c-a34f-4bdd-a04b-167d64b6c349","Type":"ContainerDied","Data":"e4b619d767fef70a1d657197b3dd40881535eb7041d55b33e17986c9f43be5ce"}
Jul 10 19:29:33.411819 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.411398    2558 scope.go:117] "RemoveContainer" containerID="e6c70b83e084aa8e2ea9aa6b44b18867b9dd81577d90c69a602b1634144702e7"
Jul 10 19:29:33.422153 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.422135    2558 scope.go:117] "RemoveContainer" containerID="ea0cfb87b169667f6cbb9510fd5eb5be09fc04fd17f170b3734e6c486ab074a0"
Jul 10 19:29:33.430150 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.430057    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"]
Jul 10 19:29:33.434322 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.433573    2558 scope.go:117] "RemoveContainer" containerID="99200cfa4b2069fe67fd3a1d6b338670976894daf10e0bf0f1fe9cd7405aa6f2"
Jul 10 19:29:33.434322 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:33.433969    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-7c8bd75989-2r4dn"]
Jul 10 19:29:35.259576 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:29:35.259341    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" path="/var/lib/kubelet/pods/435fbc3c-a34f-4bdd-a04b-167d64b6c349/volumes"
Jul 10 19:31:00.776850 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:00.776811    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerStarted","Data":"0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac"}
Jul 10 19:31:00.777163 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:00.776898    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerStarted","Data":"ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758"}
Jul 10 19:31:00.777163 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:00.776931    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:31:00.777163 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:00.776958    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:31:00.778180 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:00.778156    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 10 19:31:00.797029 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:00.796951    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" podStartSLOduration=1.765047525 podStartE2EDuration="1m58.796936655s" totalImagesPullingTime="1m53.275762696s" totalInitContainerRuntime="3.756126434s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:00.795495424 +0000 UTC m=+3106.057160199" watchObservedRunningTime="2026-07-10 19:31:00.796936655 +0000 UTC m=+3106.058601428"
Jul 10 19:31:01.780514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:01.780472    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 10 19:31:06.784904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:06.784878    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:31:06.785466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:06.785441    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Jul 10 19:31:16.786517 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:16.786492    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:31:23.873475 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873435    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"]
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873693    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kserve-container"
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873703    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kserve-container"
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873719    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="storage-initializer"
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873724    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="storage-initializer"
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873734    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy"
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873762    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy"
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873829    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kube-rbac-proxy"
Jul 10 19:31:23.873981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.873840    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="435fbc3c-a34f-4bdd-a04b-167d64b6c349" containerName="kserve-container"
Jul 10 19:31:23.876701 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.876675    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:23.877590 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.877568    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"]
Jul 10 19:31:23.877813 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.877774    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kserve-container" containerID="cri-o://ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758" gracePeriod=30
Jul 10 19:31:23.877929 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.877819    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kube-rbac-proxy" containerID="cri-o://0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac" gracePeriod=30
Jul 10 19:31:23.880655 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.880636    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 10 19:31:23.880655 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.880648    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 10 19:31:23.890156 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.890136    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"]
Jul 10 19:31:23.964614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.964585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173e0e8d-8d14-45c7-8218-f590a6e67690-proxy-tls\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:23.964779 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.964627    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173e0e8d-8d14-45c7-8218-f590a6e67690-kserve-provision-location\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:23.964779 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.964657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173e0e8d-8d14-45c7-8218-f590a6e67690-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:23.964779 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:23.964682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m5bx\" (UniqueName: \"kubernetes.io/projected/173e0e8d-8d14-45c7-8218-f590a6e67690-kube-api-access-5m5bx\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.065233 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.065210    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173e0e8d-8d14-45c7-8218-f590a6e67690-proxy-tls\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.065355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.065243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173e0e8d-8d14-45c7-8218-f590a6e67690-kserve-provision-location\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.065355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.065263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173e0e8d-8d14-45c7-8218-f590a6e67690-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.065355 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.065280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m5bx\" (UniqueName: \"kubernetes.io/projected/173e0e8d-8d14-45c7-8218-f590a6e67690-kube-api-access-5m5bx\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.065664 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.065642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173e0e8d-8d14-45c7-8218-f590a6e67690-kserve-provision-location\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.066043 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.066023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173e0e8d-8d14-45c7-8218-f590a6e67690-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.067503 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.067483    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173e0e8d-8d14-45c7-8218-f590a6e67690-proxy-tls\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.073462 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.073439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m5bx\" (UniqueName: \"kubernetes.io/projected/173e0e8d-8d14-45c7-8218-f590a6e67690-kube-api-access-5m5bx\") pod \"isvc-xgboost-predictor-77cc748fc5-wzqwb\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.195221 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.195155    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:24.321818 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.321794    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"]
Jul 10 19:31:24.324413 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:31:24.324389    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173e0e8d_8d14_45c7_8218_f590a6e67690.slice/crio-a09d6e793043f06a2651d72438e1c2f0c79fd1a4c2b60476af8235119fad495e WatchSource:0}: Error finding container a09d6e793043f06a2651d72438e1c2f0c79fd1a4c2b60476af8235119fad495e: Status 404 returned error can't find the container with id a09d6e793043f06a2651d72438e1c2f0c79fd1a4c2b60476af8235119fad495e
Jul 10 19:31:24.862345 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.862304    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerID="0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac" exitCode=2
Jul 10 19:31:24.862518 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.862385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerDied","Data":"0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac"}
Jul 10 19:31:24.863861 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.863837    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerStarted","Data":"3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28"}
Jul 10 19:31:24.863973 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:24.863892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerStarted","Data":"a09d6e793043f06a2651d72438e1c2f0c79fd1a4c2b60476af8235119fad495e"}
Jul 10 19:31:26.350400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.350378    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:31:26.482592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.482528    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kserve-provision-location\") pod \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") "
Jul 10 19:31:26.482592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.482577    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-isvc-triton-kube-rbac-proxy-sar-config\") pod \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") "
Jul 10 19:31:26.482835 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.482605    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tlz4\" (UniqueName: \"kubernetes.io/projected/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kube-api-access-5tlz4\") pod \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") "
Jul 10 19:31:26.482835 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.482629    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-proxy-tls\") pod \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\" (UID: \"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f\") "
Jul 10 19:31:26.482983 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.482962    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" (UID: "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:31:26.482983 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.482968    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-isvc-triton-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-triton-kube-rbac-proxy-sar-config") pod "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" (UID: "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f"). InnerVolumeSpecName "isvc-triton-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:31:26.484627 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.484607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" (UID: "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:31:26.484701 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.484652    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kube-api-access-5tlz4" (OuterVolumeSpecName: "kube-api-access-5tlz4") pod "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" (UID: "6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f"). InnerVolumeSpecName "kube-api-access-5tlz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:31:26.583333 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.583303    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-triton-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-isvc-triton-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:31:26.583333 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.583329    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlz4\" (UniqueName: \"kubernetes.io/projected/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kube-api-access-5tlz4\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:31:26.583333 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.583339    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:31:26.583547 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.583348    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:31:26.871444 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.871404    2558 generic.go:356] "Generic (PLEG): container finished" podID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerID="ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758" exitCode=0
Jul 10 19:31:26.871603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.871469    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"
Jul 10 19:31:26.871603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.871484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerDied","Data":"ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758"}
Jul 10 19:31:26.871603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.871541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb" event={"ID":"6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f","Type":"ContainerDied","Data":"fead6b248ec4e2ef54f1e971239d8352caf09a06c1650e7d9e7869f8d270abc2"}
Jul 10 19:31:26.871603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.871569    2558 scope.go:117] "RemoveContainer" containerID="0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac"
Jul 10 19:31:26.879609 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.879594    2558 scope.go:117] "RemoveContainer" containerID="ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758"
Jul 10 19:31:26.886638 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.886622    2558 scope.go:117] "RemoveContainer" containerID="09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42"
Jul 10 19:31:26.892556 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.892529    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"]
Jul 10 19:31:26.894819 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.894794    2558 scope.go:117] "RemoveContainer" containerID="0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac"
Jul 10 19:31:26.895104 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:31:26.895083    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac\": container with ID starting with 0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac not found: ID does not exist" containerID="0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac"
Jul 10 19:31:26.895167 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.895112    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac"} err="failed to get container status \"0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac\": rpc error: code = NotFound desc = could not find container \"0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac\": container with ID starting with 0d41f22fa111dde47769c87a3f2ce7f91f9aedfc186d87fd13088be1b2b170ac not found: ID does not exist"
Jul 10 19:31:26.895167 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.895131    2558 scope.go:117] "RemoveContainer" containerID="ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758"
Jul 10 19:31:26.895363 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:31:26.895348    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758\": container with ID starting with ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758 not found: ID does not exist" containerID="ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758"
Jul 10 19:31:26.895415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.895369    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758"} err="failed to get container status \"ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758\": rpc error: code = NotFound desc = could not find container \"ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758\": container with ID starting with ad0822815c7e302488440ebf85acfb3839a781543c119e8b90380b3409ae4758 not found: ID does not exist"
Jul 10 19:31:26.895415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.895385    2558 scope.go:117] "RemoveContainer" containerID="09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42"
Jul 10 19:31:26.895611 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:31:26.895597    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42\": container with ID starting with 09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42 not found: ID does not exist" containerID="09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42"
Jul 10 19:31:26.895649 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.895615    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42"} err="failed to get container status \"09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42\": rpc error: code = NotFound desc = could not find container \"09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42\": container with ID starting with 09fad7e28377d99de3b090fe19addc0f756515fc9eec0c979f477b7a14659e42 not found: ID does not exist"
Jul 10 19:31:26.895826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:26.895809    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-6cfd998464-2ncxb"]
Jul 10 19:31:27.257449 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:27.257424    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" path="/var/lib/kubelet/pods/6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f/volumes"
Jul 10 19:31:28.881045 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:28.881012    2558 generic.go:356] "Generic (PLEG): container finished" podID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerID="3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28" exitCode=0
Jul 10 19:31:28.881454 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:28.881066    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerDied","Data":"3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28"}
Jul 10 19:31:49.972155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:49.972123    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerStarted","Data":"31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f"}
Jul 10 19:31:49.972561 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:49.972185    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerStarted","Data":"b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab"}
Jul 10 19:31:49.972561 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:49.972373    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:49.990349 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:49.990303    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podStartSLOduration=3.029093873 podStartE2EDuration="26.990288809s" totalImagesPullingTime="20.042355797s" totalInitContainerRuntime="3.918839139s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:49.989262592 +0000 UTC m=+3155.250927365" watchObservedRunningTime="2026-07-10 19:31:49.990288809 +0000 UTC m=+3155.251953582"
Jul 10 19:31:50.976122 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:50.976089    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:50.977139 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:50.977112    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:31:51.979177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:51.979138    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:31:56.983538 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:56.983511    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:31:56.983983 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:31:56.983931    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:32:06.984491 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:06.984447    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:32:16.984266 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:16.984225    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:32:26.984924 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:26.984888    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:32:36.984549 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:36.984509    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:32:46.984857 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:46.984827    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:32:54.058673 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.058619    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"]
Jul 10 19:32:54.059141 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.059006    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" containerID="cri-o://b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab" gracePeriod=30
Jul 10 19:32:54.059141 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.059039    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kube-rbac-proxy" containerID="cri-o://31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f" gracePeriod=30
Jul 10 19:32:54.078867 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.078839    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"]
Jul 10 19:32:54.079225 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079207    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kube-rbac-proxy"
Jul 10 19:32:54.079290 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079229    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kube-rbac-proxy"
Jul 10 19:32:54.079290 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079254    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="storage-initializer"
Jul 10 19:32:54.079290 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079262    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="storage-initializer"
Jul 10 19:32:54.079290 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079272    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kserve-container"
Jul 10 19:32:54.079290 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079280    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kserve-container"
Jul 10 19:32:54.079463 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079344    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kube-rbac-proxy"
Jul 10 19:32:54.079463 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.079357    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6dfcf8-3934-4a28-ac84-b4b8126bfc2f" containerName="kserve-container"
Jul 10 19:32:54.081920 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.081903    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.084015 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.083998    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 10 19:32:54.084156 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.084129    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 10 19:32:54.091948 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.091926    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"]
Jul 10 19:32:54.191316 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.191293    2558 generic.go:356] "Generic (PLEG): container finished" podID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerID="31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f" exitCode=2
Jul 10 19:32:54.191438 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.191361    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerDied","Data":"31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f"}
Jul 10 19:32:54.217381 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.217361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f519a1f-7a51-494b-9dc0-3d057129962a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.217493 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.217395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f519a1f-7a51-494b-9dc0-3d057129962a-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.217493 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.217421    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f519a1f-7a51-494b-9dc0-3d057129962a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.217493 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.217450    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmjgv\" (UniqueName: \"kubernetes.io/projected/4f519a1f-7a51-494b-9dc0-3d057129962a-kube-api-access-hmjgv\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.318354 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.318289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f519a1f-7a51-494b-9dc0-3d057129962a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.318354 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.318327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f519a1f-7a51-494b-9dc0-3d057129962a-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.318526 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.318355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f519a1f-7a51-494b-9dc0-3d057129962a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.318526 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.318384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmjgv\" (UniqueName: \"kubernetes.io/projected/4f519a1f-7a51-494b-9dc0-3d057129962a-kube-api-access-hmjgv\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.318762 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.318725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f519a1f-7a51-494b-9dc0-3d057129962a-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.319037 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.319014    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f519a1f-7a51-494b-9dc0-3d057129962a-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.320680 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.320654    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f519a1f-7a51-494b-9dc0-3d057129962a-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.326115 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.326094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmjgv\" (UniqueName: \"kubernetes.io/projected/4f519a1f-7a51-494b-9dc0-3d057129962a-kube-api-access-hmjgv\") pod \"isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.392023 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.391992    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:32:54.522185 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:54.522156    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"]
Jul 10 19:32:54.525537 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:32:54.525511    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f519a1f_7a51_494b_9dc0_3d057129962a.slice/crio-01aed378e38b3cf9b49843ccf9675339472bb5dae5f6b08ce372060160db1cfc WatchSource:0}: Error finding container 01aed378e38b3cf9b49843ccf9675339472bb5dae5f6b08ce372060160db1cfc: Status 404 returned error can't find the container with id 01aed378e38b3cf9b49843ccf9675339472bb5dae5f6b08ce372060160db1cfc
Jul 10 19:32:55.196984 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:55.196950    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerStarted","Data":"51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443"}
Jul 10 19:32:55.197406 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:55.197004    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerStarted","Data":"01aed378e38b3cf9b49843ccf9675339472bb5dae5f6b08ce372060160db1cfc"}
Jul 10 19:32:56.980184 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:56.980142    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Jul 10 19:32:56.984501 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:56.984476    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Jul 10 19:32:57.509446 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.509423    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:32:57.643084 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.643020    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173e0e8d-8d14-45c7-8218-f590a6e67690-proxy-tls\") pod \"173e0e8d-8d14-45c7-8218-f590a6e67690\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") "
Jul 10 19:32:57.643084 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.643054    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173e0e8d-8d14-45c7-8218-f590a6e67690-kserve-provision-location\") pod \"173e0e8d-8d14-45c7-8218-f590a6e67690\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") "
Jul 10 19:32:57.643400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.643089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173e0e8d-8d14-45c7-8218-f590a6e67690-isvc-xgboost-kube-rbac-proxy-sar-config\") pod \"173e0e8d-8d14-45c7-8218-f590a6e67690\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") "
Jul 10 19:32:57.643400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.643112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m5bx\" (UniqueName: \"kubernetes.io/projected/173e0e8d-8d14-45c7-8218-f590a6e67690-kube-api-access-5m5bx\") pod \"173e0e8d-8d14-45c7-8218-f590a6e67690\" (UID: \"173e0e8d-8d14-45c7-8218-f590a6e67690\") "
Jul 10 19:32:57.643400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.643342    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173e0e8d-8d14-45c7-8218-f590a6e67690-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "173e0e8d-8d14-45c7-8218-f590a6e67690" (UID: "173e0e8d-8d14-45c7-8218-f590a6e67690"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:32:57.643524 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.643436    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173e0e8d-8d14-45c7-8218-f590a6e67690-isvc-xgboost-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-kube-rbac-proxy-sar-config") pod "173e0e8d-8d14-45c7-8218-f590a6e67690" (UID: "173e0e8d-8d14-45c7-8218-f590a6e67690"). InnerVolumeSpecName "isvc-xgboost-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:32:57.645091 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.645065    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173e0e8d-8d14-45c7-8218-f590a6e67690-kube-api-access-5m5bx" (OuterVolumeSpecName: "kube-api-access-5m5bx") pod "173e0e8d-8d14-45c7-8218-f590a6e67690" (UID: "173e0e8d-8d14-45c7-8218-f590a6e67690"). InnerVolumeSpecName "kube-api-access-5m5bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:32:57.645197 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.645164    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173e0e8d-8d14-45c7-8218-f590a6e67690-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "173e0e8d-8d14-45c7-8218-f590a6e67690" (UID: "173e0e8d-8d14-45c7-8218-f590a6e67690"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:32:57.744313 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.744286    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173e0e8d-8d14-45c7-8218-f590a6e67690-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:32:57.744313 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.744308    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173e0e8d-8d14-45c7-8218-f590a6e67690-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:32:57.744313 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.744318    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173e0e8d-8d14-45c7-8218-f590a6e67690-isvc-xgboost-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:32:57.744498 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:57.744328    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m5bx\" (UniqueName: \"kubernetes.io/projected/173e0e8d-8d14-45c7-8218-f590a6e67690-kube-api-access-5m5bx\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:32:58.208668 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.208636    2558 generic.go:356] "Generic (PLEG): container finished" podID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerID="b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab" exitCode=0
Jul 10 19:32:58.209096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.208687    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerDied","Data":"b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab"}
Jul 10 19:32:58.209096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.208733    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb" event={"ID":"173e0e8d-8d14-45c7-8218-f590a6e67690","Type":"ContainerDied","Data":"a09d6e793043f06a2651d72438e1c2f0c79fd1a4c2b60476af8235119fad495e"}
Jul 10 19:32:58.209096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.208690    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"
Jul 10 19:32:58.209096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.208775    2558 scope.go:117] "RemoveContainer" containerID="31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f"
Jul 10 19:32:58.217667 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.217649    2558 scope.go:117] "RemoveContainer" containerID="b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab"
Jul 10 19:32:58.224715 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.224699    2558 scope.go:117] "RemoveContainer" containerID="3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28"
Jul 10 19:32:58.231065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.231045    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"]
Jul 10 19:32:58.231843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.231819    2558 scope.go:117] "RemoveContainer" containerID="31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f"
Jul 10 19:32:58.232058 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:32:58.232042    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f\": container with ID starting with 31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f not found: ID does not exist" containerID="31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f"
Jul 10 19:32:58.232117 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.232064    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f"} err="failed to get container status \"31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f\": rpc error: code = NotFound desc = could not find container \"31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f\": container with ID starting with 31e84c8008485b327c5a8535513037b5723ec2bbbcbcd5e73609013f1bc5d71f not found: ID does not exist"
Jul 10 19:32:58.232117 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.232080    2558 scope.go:117] "RemoveContainer" containerID="b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab"
Jul 10 19:32:58.232321 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:32:58.232305    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab\": container with ID starting with b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab not found: ID does not exist" containerID="b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab"
Jul 10 19:32:58.232384 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.232330    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab"} err="failed to get container status \"b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab\": rpc error: code = NotFound desc = could not find container \"b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab\": container with ID starting with b5618222ffcafc7c6551469fcc08af5837e7435ee81543c28037b7dbf9693bab not found: ID does not exist"
Jul 10 19:32:58.232384 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.232353    2558 scope.go:117] "RemoveContainer" containerID="3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28"
Jul 10 19:32:58.232559 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:32:58.232544    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28\": container with ID starting with 3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28 not found: ID does not exist" containerID="3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28"
Jul 10 19:32:58.232613 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.232567    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28"} err="failed to get container status \"3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28\": rpc error: code = NotFound desc = could not find container \"3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28\": container with ID starting with 3f1699f7ad0099a51fb34dd6ac5e9cd2146e2fed44154214c9f43e6ac1041b28 not found: ID does not exist"
Jul 10 19:32:58.234970 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:58.234952    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-77cc748fc5-wzqwb"]
Jul 10 19:32:59.213961 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:59.213924    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerID="51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443" exitCode=0
Jul 10 19:32:59.214447 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:59.213985    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerDied","Data":"51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443"}
Jul 10 19:32:59.258558 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:32:59.258518    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" path="/var/lib/kubelet/pods/173e0e8d-8d14-45c7-8218-f590a6e67690/volumes"
Jul 10 19:33:00.219166 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:00.219135    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerStarted","Data":"35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e"}
Jul 10 19:33:00.219546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:00.219216    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerStarted","Data":"866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1"}
Jul 10 19:33:00.219546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:00.219403    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:33:00.240082 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:00.240035    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" podStartSLOduration=2.2990101960000002 podStartE2EDuration="6.240018222s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941008026s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:33:00.239366252 +0000 UTC m=+3225.501031036" watchObservedRunningTime="2026-07-10 19:33:00.240018222 +0000 UTC m=+3225.501682995"
Jul 10 19:33:01.223122 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:01.223097    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:33:07.232758 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:07.232606    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:33:37.237583 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:37.237548    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:33:44.151906 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.151877    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"]
Jul 10 19:33:44.152339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.152097    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kserve-container" containerID="cri-o://866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1" gracePeriod=30
Jul 10 19:33:44.152339 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.152145    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kube-rbac-proxy" containerID="cri-o://35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e" gracePeriod=30
Jul 10 19:33:44.237236 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237194    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"]
Jul 10 19:33:44.237540 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237524    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kube-rbac-proxy"
Jul 10 19:33:44.237631 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237543    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kube-rbac-proxy"
Jul 10 19:33:44.237631 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237558    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="storage-initializer"
Jul 10 19:33:44.237631 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237566    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="storage-initializer"
Jul 10 19:33:44.237631 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237575    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container"
Jul 10 19:33:44.237631 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237583    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container"
Jul 10 19:33:44.237950 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237649    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kube-rbac-proxy"
Jul 10 19:33:44.237950 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.237666    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="173e0e8d-8d14-45c7-8218-f590a6e67690" containerName="kserve-container"
Jul 10 19:33:44.240202 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.240180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.242548 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.242525    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 10 19:33:44.242650 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.242525    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 10 19:33:44.250760 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.250718    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"]
Jul 10 19:33:44.274243 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.274221    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.274352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.274250    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm4s\" (UniqueName: \"kubernetes.io/projected/8d301a3f-814f-471e-9258-8939c0c7edb7-kube-api-access-mfm4s\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.274352 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.274325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d301a3f-814f-471e-9258-8939c0c7edb7-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.274434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.274350    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d301a3f-814f-471e-9258-8939c0c7edb7-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.371140 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.371108    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerID="35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e" exitCode=2
Jul 10 19:33:44.371315 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.371186    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerDied","Data":"35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e"}
Jul 10 19:33:44.375459 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.375439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d301a3f-814f-471e-9258-8939c0c7edb7-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.375585 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.375470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d301a3f-814f-471e-9258-8939c0c7edb7-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.375585 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.375524    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.375585 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.375547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm4s\" (UniqueName: \"kubernetes.io/projected/8d301a3f-814f-471e-9258-8939c0c7edb7-kube-api-access-mfm4s\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.375770 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:33:44.375716    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 10 19:33:44.375837 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:33:44.375825    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls podName:8d301a3f-814f-471e-9258-8939c0c7edb7 nodeName:}" failed. No retries permitted until 2026-07-10 19:33:44.875800705 +0000 UTC m=+3270.137465468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls") pod "xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" (UID: "8d301a3f-814f-471e-9258-8939c0c7edb7") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 10 19:33:44.375972 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.375956    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d301a3f-814f-471e-9258-8939c0c7edb7-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.376184 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.376167    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d301a3f-814f-471e-9258-8939c0c7edb7-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.383915 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.383893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm4s\" (UniqueName: \"kubernetes.io/projected/8d301a3f-814f-471e-9258-8939c0c7edb7-kube-api-access-mfm4s\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.878556 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.878526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:44.880879 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:44.880856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-849b7f99bc-6g47b\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:45.149784 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:45.149690    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:45.275193 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:45.275164    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"]
Jul 10 19:33:45.278068 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:33:45.278039    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d301a3f_814f_471e_9258_8939c0c7edb7.slice/crio-03477af4ced5c0892f36ef70d47019f860f424863812df751381737125c573b1 WatchSource:0}: Error finding container 03477af4ced5c0892f36ef70d47019f860f424863812df751381737125c573b1: Status 404 returned error can't find the container with id 03477af4ced5c0892f36ef70d47019f860f424863812df751381737125c573b1
Jul 10 19:33:45.376125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:45.376096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerStarted","Data":"8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a"}
Jul 10 19:33:45.376250 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:45.376152    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerStarted","Data":"03477af4ced5c0892f36ef70d47019f860f424863812df751381737125c573b1"}
Jul 10 19:33:47.227072 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:47.227033    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Jul 10 19:33:47.233517 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:47.233489    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.55:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.55:8080: connect: connection refused"
Jul 10 19:33:49.390570 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:49.390487    2558 generic.go:356] "Generic (PLEG): container finished" podID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerID="8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a" exitCode=0
Jul 10 19:33:49.390996 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:49.390565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerDied","Data":"8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a"}
Jul 10 19:33:50.285237 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.285217    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:33:50.312972 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.312943    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f519a1f-7a51-494b-9dc0-3d057129962a-kserve-provision-location\") pod \"4f519a1f-7a51-494b-9dc0-3d057129962a\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") "
Jul 10 19:33:50.313096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.312987    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmjgv\" (UniqueName: \"kubernetes.io/projected/4f519a1f-7a51-494b-9dc0-3d057129962a-kube-api-access-hmjgv\") pod \"4f519a1f-7a51-494b-9dc0-3d057129962a\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") "
Jul 10 19:33:50.313096 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.313019    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f519a1f-7a51-494b-9dc0-3d057129962a-proxy-tls\") pod \"4f519a1f-7a51-494b-9dc0-3d057129962a\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") "
Jul 10 19:33:50.313231 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.313105    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f519a1f-7a51-494b-9dc0-3d057129962a-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"4f519a1f-7a51-494b-9dc0-3d057129962a\" (UID: \"4f519a1f-7a51-494b-9dc0-3d057129962a\") "
Jul 10 19:33:50.313294 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.313242    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f519a1f-7a51-494b-9dc0-3d057129962a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f519a1f-7a51-494b-9dc0-3d057129962a" (UID: "4f519a1f-7a51-494b-9dc0-3d057129962a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:33:50.313350 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.313314    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f519a1f-7a51-494b-9dc0-3d057129962a-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:33:50.313555 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.313528    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f519a1f-7a51-494b-9dc0-3d057129962a-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "4f519a1f-7a51-494b-9dc0-3d057129962a" (UID: "4f519a1f-7a51-494b-9dc0-3d057129962a"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:33:50.315155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.315131    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f519a1f-7a51-494b-9dc0-3d057129962a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f519a1f-7a51-494b-9dc0-3d057129962a" (UID: "4f519a1f-7a51-494b-9dc0-3d057129962a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:33:50.315234 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.315189    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f519a1f-7a51-494b-9dc0-3d057129962a-kube-api-access-hmjgv" (OuterVolumeSpecName: "kube-api-access-hmjgv") pod "4f519a1f-7a51-494b-9dc0-3d057129962a" (UID: "4f519a1f-7a51-494b-9dc0-3d057129962a"). InnerVolumeSpecName "kube-api-access-hmjgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:33:50.396028 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.395960    2558 generic.go:356] "Generic (PLEG): container finished" podID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerID="866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1" exitCode=0
Jul 10 19:33:50.396028 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.396020    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"
Jul 10 19:33:50.396450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.396046    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerDied","Data":"866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1"}
Jul 10 19:33:50.396450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.396119    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f" event={"ID":"4f519a1f-7a51-494b-9dc0-3d057129962a","Type":"ContainerDied","Data":"01aed378e38b3cf9b49843ccf9675339472bb5dae5f6b08ce372060160db1cfc"}
Jul 10 19:33:50.396450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.396155    2558 scope.go:117] "RemoveContainer" containerID="35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e"
Jul 10 19:33:50.398495 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.398473    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerStarted","Data":"56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7"}
Jul 10 19:33:50.398606 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.398524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerStarted","Data":"f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6"}
Jul 10 19:33:50.398723 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.398709    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:50.398809 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.398784    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:33:50.405235 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.405218    2558 scope.go:117] "RemoveContainer" containerID="866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1"
Jul 10 19:33:50.412427 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.412410    2558 scope.go:117] "RemoveContainer" containerID="51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443"
Jul 10 19:33:50.414582 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.414553    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f519a1f-7a51-494b-9dc0-3d057129962a-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:33:50.414666 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.414581    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmjgv\" (UniqueName: \"kubernetes.io/projected/4f519a1f-7a51-494b-9dc0-3d057129962a-kube-api-access-hmjgv\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:33:50.414666 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.414596    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f519a1f-7a51-494b-9dc0-3d057129962a-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:33:50.419228 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.419190    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" podStartSLOduration=2.69857637 podStartE2EDuration="6.419175893s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.720599523s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:33:50.417957358 +0000 UTC m=+3275.679622131" watchObservedRunningTime="2026-07-10 19:33:50.419175893 +0000 UTC m=+3275.680840665"
Jul 10 19:33:50.419644 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.419625    2558 scope.go:117] "RemoveContainer" containerID="35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e"
Jul 10 19:33:50.419917 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:33:50.419889    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e\": container with ID starting with 35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e not found: ID does not exist" containerID="35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e"
Jul 10 19:33:50.419980 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.419918    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e"} err="failed to get container status \"35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e\": rpc error: code = NotFound desc = could not find container \"35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e\": container with ID starting with 35580f5409e49fba075cb451b504e00b3d1e81e3b44ccfd7cedcaf85b45c6e4e not found: ID does not exist"
Jul 10 19:33:50.419980 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.419938    2558 scope.go:117] "RemoveContainer" containerID="866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1"
Jul 10 19:33:50.420161 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:33:50.420142    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1\": container with ID starting with 866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1 not found: ID does not exist" containerID="866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1"
Jul 10 19:33:50.420200 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.420168    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1"} err="failed to get container status \"866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1\": rpc error: code = NotFound desc = could not find container \"866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1\": container with ID starting with 866093cd66954b9c35394cee2316fc14ae926b62c1a381a462300e4b85ff7bc1 not found: ID does not exist"
Jul 10 19:33:50.420200 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.420185    2558 scope.go:117] "RemoveContainer" containerID="51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443"
Jul 10 19:33:50.420429 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:33:50.420413    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443\": container with ID starting with 51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443 not found: ID does not exist" containerID="51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443"
Jul 10 19:33:50.420474 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.420435    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443"} err="failed to get container status \"51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443\": rpc error: code = NotFound desc = could not find container \"51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443\": container with ID starting with 51b1ba0831f682567d400c7ae4979ce55951235133d1a59ab965128faaaa3443 not found: ID does not exist"
Jul 10 19:33:50.430529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.430508    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"]
Jul 10 19:33:50.434393 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:50.434373    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-69ddd65786-lt88f"]
Jul 10 19:33:51.257186 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:51.257153    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" path="/var/lib/kubelet/pods/4f519a1f-7a51-494b-9dc0-3d057129962a/volumes"
Jul 10 19:33:56.408169 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:33:56.408142    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:34:15.437256 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:15.437231    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:34:18.677803 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:18.677773    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:34:26.412234 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:26.412206    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:34:34.339820 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.339783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"]
Jul 10 19:34:34.340222 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.340005    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kserve-container" containerID="cri-o://f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6" gracePeriod=30
Jul 10 19:34:34.340222 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.340059    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kube-rbac-proxy" containerID="cri-o://56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7" gracePeriod=30
Jul 10 19:34:34.406299 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406268    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"]
Jul 10 19:34:34.406530 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406517    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kserve-container"
Jul 10 19:34:34.406577 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406532    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kserve-container"
Jul 10 19:34:34.406577 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406546    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kube-rbac-proxy"
Jul 10 19:34:34.406577 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406552    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kube-rbac-proxy"
Jul 10 19:34:34.406577 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406564    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="storage-initializer"
Jul 10 19:34:34.406577 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406570    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="storage-initializer"
Jul 10 19:34:34.406985 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406611    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kube-rbac-proxy"
Jul 10 19:34:34.406985 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.406620    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f519a1f-7a51-494b-9dc0-3d057129962a" containerName="kserve-container"
Jul 10 19:34:34.409251 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.409226    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.411593 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.411574    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:34:34.411699 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.411649    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 10 19:34:34.420573 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.420552    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"]
Jul 10 19:34:34.519451 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.519417    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c8be97-9fba-4f2a-bb30-836cbf3ce384-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.519606 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.519480    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrwmq\" (UniqueName: \"kubernetes.io/projected/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kube-api-access-wrwmq\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.519606 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.519556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.519702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.519627    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c8be97-9fba-4f2a-bb30-836cbf3ce384-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.552460 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.552431    2558 generic.go:356] "Generic (PLEG): container finished" podID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerID="56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7" exitCode=2
Jul 10 19:34:34.552583 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.552513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerDied","Data":"56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7"}
Jul 10 19:34:34.621065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.620995    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c8be97-9fba-4f2a-bb30-836cbf3ce384-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.621065 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.621051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c8be97-9fba-4f2a-bb30-836cbf3ce384-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.621267 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.621098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrwmq\" (UniqueName: \"kubernetes.io/projected/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kube-api-access-wrwmq\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.621267 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.621130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.621499 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.621478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.621666 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.621645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c8be97-9fba-4f2a-bb30-836cbf3ce384-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.623551 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.623529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c8be97-9fba-4f2a-bb30-836cbf3ce384-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.629730 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.629711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrwmq\" (UniqueName: \"kubernetes.io/projected/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kube-api-access-wrwmq\") pod \"isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.720492 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.720457    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:34.841361 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:34.841314    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"]
Jul 10 19:34:34.843797 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:34:34.843769    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c8be97_9fba_4f2a_bb30_836cbf3ce384.slice/crio-7c6bc9bfbdfe8b2b021c9939b8b4e99c3047004228aebfc350e61ca181a4ee27 WatchSource:0}: Error finding container 7c6bc9bfbdfe8b2b021c9939b8b4e99c3047004228aebfc350e61ca181a4ee27: Status 404 returned error can't find the container with id 7c6bc9bfbdfe8b2b021c9939b8b4e99c3047004228aebfc350e61ca181a4ee27
Jul 10 19:34:35.557585 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:35.557553    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerStarted","Data":"913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c"}
Jul 10 19:34:35.558011 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:35.557605    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerStarted","Data":"7c6bc9bfbdfe8b2b021c9939b8b4e99c3047004228aebfc350e61ca181a4ee27"}
Jul 10 19:34:36.403526 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:36.403484    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Jul 10 19:34:38.569231 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:38.569201    2558 generic.go:356] "Generic (PLEG): container finished" podID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerID="913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c" exitCode=0
Jul 10 19:34:38.569570 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:38.569275    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerDied","Data":"913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c"}
Jul 10 19:34:39.575437 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:39.575406    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerStarted","Data":"39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309"}
Jul 10 19:34:39.575861 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:39.575464    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerStarted","Data":"5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630"}
Jul 10 19:34:39.575861 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:39.575776    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:39.575981 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:39.575935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:39.577185 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:39.577158    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:34:39.595754 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:39.595707    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podStartSLOduration=2.025846548 podStartE2EDuration="5.595692696s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.569846148s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:39.59404935 +0000 UTC m=+3324.855714122" watchObservedRunningTime="2026-07-10 19:34:39.595692696 +0000 UTC m=+3324.857357468"
Jul 10 19:34:40.371773 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.371737    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:34:40.461441 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.461354    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d301a3f-814f-471e-9258-8939c0c7edb7-kserve-provision-location\") pod \"8d301a3f-814f-471e-9258-8939c0c7edb7\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") "
Jul 10 19:34:40.461441 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.461428    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfm4s\" (UniqueName: \"kubernetes.io/projected/8d301a3f-814f-471e-9258-8939c0c7edb7-kube-api-access-mfm4s\") pod \"8d301a3f-814f-471e-9258-8939c0c7edb7\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") "
Jul 10 19:34:40.461630 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.461459    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d301a3f-814f-471e-9258-8939c0c7edb7-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"8d301a3f-814f-471e-9258-8939c0c7edb7\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") "
Jul 10 19:34:40.461630 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.461482    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls\") pod \"8d301a3f-814f-471e-9258-8939c0c7edb7\" (UID: \"8d301a3f-814f-471e-9258-8939c0c7edb7\") "
Jul 10 19:34:40.461731 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.461708    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d301a3f-814f-471e-9258-8939c0c7edb7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d301a3f-814f-471e-9258-8939c0c7edb7" (UID: "8d301a3f-814f-471e-9258-8939c0c7edb7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:34:40.461827 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.461807    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d301a3f-814f-471e-9258-8939c0c7edb7-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "8d301a3f-814f-471e-9258-8939c0c7edb7" (UID: "8d301a3f-814f-471e-9258-8939c0c7edb7"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:34:40.463597 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.463559    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d301a3f-814f-471e-9258-8939c0c7edb7" (UID: "8d301a3f-814f-471e-9258-8939c0c7edb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:34:40.463597 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.463592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d301a3f-814f-471e-9258-8939c0c7edb7-kube-api-access-mfm4s" (OuterVolumeSpecName: "kube-api-access-mfm4s") pod "8d301a3f-814f-471e-9258-8939c0c7edb7" (UID: "8d301a3f-814f-471e-9258-8939c0c7edb7"). InnerVolumeSpecName "kube-api-access-mfm4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:34:40.562117 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.562071    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d301a3f-814f-471e-9258-8939c0c7edb7-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:34:40.562117 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.562110    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfm4s\" (UniqueName: \"kubernetes.io/projected/8d301a3f-814f-471e-9258-8939c0c7edb7-kube-api-access-mfm4s\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:34:40.562117 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.562122    2558 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d301a3f-814f-471e-9258-8939c0c7edb7-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:34:40.562117 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.562132    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d301a3f-814f-471e-9258-8939c0c7edb7-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:34:40.580723 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.580693    2558 generic.go:356] "Generic (PLEG): container finished" podID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerID="f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6" exitCode=0
Jul 10 19:34:40.581088 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.580769    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"
Jul 10 19:34:40.581088 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.580773    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerDied","Data":"f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6"}
Jul 10 19:34:40.581088 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.580826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b" event={"ID":"8d301a3f-814f-471e-9258-8939c0c7edb7","Type":"ContainerDied","Data":"03477af4ced5c0892f36ef70d47019f860f424863812df751381737125c573b1"}
Jul 10 19:34:40.581088 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.580854    2558 scope.go:117] "RemoveContainer" containerID="56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7"
Jul 10 19:34:40.581366 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.581342    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:34:40.589506 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.589489    2558 scope.go:117] "RemoveContainer" containerID="f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6"
Jul 10 19:34:40.596447 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.596430    2558 scope.go:117] "RemoveContainer" containerID="8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a"
Jul 10 19:34:40.602701 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.602679    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"]
Jul 10 19:34:40.603854 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.603833    2558 scope.go:117] "RemoveContainer" containerID="56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7"
Jul 10 19:34:40.604118 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:34:40.604095    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7\": container with ID starting with 56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7 not found: ID does not exist" containerID="56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7"
Jul 10 19:34:40.604182 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.604129    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7"} err="failed to get container status \"56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7\": rpc error: code = NotFound desc = could not find container \"56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7\": container with ID starting with 56e8dbfbd4aaf57386e07b0824dd751894436e6455ae22dee4f2897a43cbf7d7 not found: ID does not exist"
Jul 10 19:34:40.604182 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.604149    2558 scope.go:117] "RemoveContainer" containerID="f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6"
Jul 10 19:34:40.604469 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:34:40.604442    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6\": container with ID starting with f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6 not found: ID does not exist" containerID="f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6"
Jul 10 19:34:40.604564 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.604479    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6"} err="failed to get container status \"f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6\": rpc error: code = NotFound desc = could not find container \"f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6\": container with ID starting with f774653fd9d732bd81bf942b109c681943808600e7124051814ae585a61131b6 not found: ID does not exist"
Jul 10 19:34:40.604564 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.604559    2558 scope.go:117] "RemoveContainer" containerID="8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a"
Jul 10 19:34:40.605332 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:34:40.605306    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a\": container with ID starting with 8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a not found: ID does not exist" containerID="8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a"
Jul 10 19:34:40.605413 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.605341    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a"} err="failed to get container status \"8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a\": rpc error: code = NotFound desc = could not find container \"8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a\": container with ID starting with 8a515fbc394f86dd530fa2292944b61fd339153015dcdf77c14b460986f2724a not found: ID does not exist"
Jul 10 19:34:40.606341 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:40.606325    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-849b7f99bc-6g47b"]
Jul 10 19:34:41.256215 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:41.256187    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" path="/var/lib/kubelet/pods/8d301a3f-814f-471e-9258-8939c0c7edb7/volumes"
Jul 10 19:34:41.585689 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:41.585604    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:34:46.590614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:46.590587    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:34:46.591071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:46.591050    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:34:56.591689 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:34:56.591652    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:35:06.591284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:06.591248    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:35:16.591586 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:16.591544    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:35:26.591039 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:26.590998    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:35:36.591888 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:36.591859    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:35:44.495707 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.495676    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"]
Jul 10 19:35:44.496151 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.495926    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" containerID="cri-o://5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630" gracePeriod=30
Jul 10 19:35:44.496151 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.495959    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kube-rbac-proxy" containerID="cri-o://39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309" gracePeriod=30
Jul 10 19:35:44.573679 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.573646    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"]
Jul 10 19:35:44.573928 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.573915    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="storage-initializer"
Jul 10 19:35:44.573978 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.573929    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="storage-initializer"
Jul 10 19:35:44.573978 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.573940    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kserve-container"
Jul 10 19:35:44.573978 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.573945    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kserve-container"
Jul 10 19:35:44.573978 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.573950    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kube-rbac-proxy"
Jul 10 19:35:44.573978 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.573956    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kube-rbac-proxy"
Jul 10 19:35:44.574131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.574004    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kube-rbac-proxy"
Jul 10 19:35:44.574131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.574011    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d301a3f-814f-471e-9258-8939c0c7edb7" containerName="kserve-container"
Jul 10 19:35:44.576984 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.576966    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.579061 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.579042    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 10 19:35:44.579230 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.579215    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 10 19:35:44.587107 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.587084    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"]
Jul 10 19:35:44.611059 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.611039    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.611171 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.611074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.611171 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.611108    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.611272 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.611175    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwlp\" (UniqueName: \"kubernetes.io/projected/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kube-api-access-rgwlp\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.712191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.712157    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.712314 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.712200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.712314 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.712239    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.712314 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.712258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwlp\" (UniqueName: \"kubernetes.io/projected/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kube-api-access-rgwlp\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.712490 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:35:44.712331    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 10 19:35:44.712490 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:35:44.712402    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls podName:c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93 nodeName:}" failed. No retries permitted until 2026-07-10 19:35:45.212379222 +0000 UTC m=+3390.474043987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" (UID: "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 10 19:35:44.712619 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.712597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.712912 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.712893    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.722516 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.722492    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwlp\" (UniqueName: \"kubernetes.io/projected/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kube-api-access-rgwlp\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:44.807571 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.807488    2558 generic.go:356] "Generic (PLEG): container finished" podID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerID="39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309" exitCode=2
Jul 10 19:35:44.807571 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:44.807538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerDied","Data":"39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309"}
Jul 10 19:35:45.214727 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:45.214640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:45.217058 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:45.217039    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:45.486661 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:45.486581    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:45.613788 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:45.613765    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"]
Jul 10 19:35:45.616091 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:35:45.616063    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32fe8bb_2cdf_4738_b8b5_c9a8f2444b93.slice/crio-417208b4a4243e412c50d31c4e6cf5a527c02bcd723047dd4c8149b45acc6771 WatchSource:0}: Error finding container 417208b4a4243e412c50d31c4e6cf5a527c02bcd723047dd4c8149b45acc6771: Status 404 returned error can't find the container with id 417208b4a4243e412c50d31c4e6cf5a527c02bcd723047dd4c8149b45acc6771
Jul 10 19:35:45.813326 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:45.813286    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerStarted","Data":"c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e"}
Jul 10 19:35:45.813499 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:45.813363    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerStarted","Data":"417208b4a4243e412c50d31c4e6cf5a527c02bcd723047dd4c8149b45acc6771"}
Jul 10 19:35:46.586793 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:46.586736    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Jul 10 19:35:46.591089 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:46.591060    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Jul 10 19:35:48.042653 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.042630    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:35:48.137085 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.137010    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c8be97-9fba-4f2a-bb30-836cbf3ce384-proxy-tls\") pod \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") "
Jul 10 19:35:48.137085 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.137064    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c8be97-9fba-4f2a-bb30-836cbf3ce384-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") pod \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") "
Jul 10 19:35:48.137244 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.137102    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrwmq\" (UniqueName: \"kubernetes.io/projected/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kube-api-access-wrwmq\") pod \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") "
Jul 10 19:35:48.137244 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.137120    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kserve-provision-location\") pod \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\" (UID: \"71c8be97-9fba-4f2a-bb30-836cbf3ce384\") "
Jul 10 19:35:48.137444 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.137423    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c8be97-9fba-4f2a-bb30-836cbf3ce384-isvc-xgboost-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-runtime-kube-rbac-proxy-sar-config") pod "71c8be97-9fba-4f2a-bb30-836cbf3ce384" (UID: "71c8be97-9fba-4f2a-bb30-836cbf3ce384"). InnerVolumeSpecName "isvc-xgboost-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:35:48.137510 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.137464    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71c8be97-9fba-4f2a-bb30-836cbf3ce384" (UID: "71c8be97-9fba-4f2a-bb30-836cbf3ce384"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:35:48.139064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.139046    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c8be97-9fba-4f2a-bb30-836cbf3ce384-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71c8be97-9fba-4f2a-bb30-836cbf3ce384" (UID: "71c8be97-9fba-4f2a-bb30-836cbf3ce384"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:35:48.139168 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.139154    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kube-api-access-wrwmq" (OuterVolumeSpecName: "kube-api-access-wrwmq") pod "71c8be97-9fba-4f2a-bb30-836cbf3ce384" (UID: "71c8be97-9fba-4f2a-bb30-836cbf3ce384"). InnerVolumeSpecName "kube-api-access-wrwmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:35:48.237918 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.237884    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71c8be97-9fba-4f2a-bb30-836cbf3ce384-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:35:48.237918 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.237916    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71c8be97-9fba-4f2a-bb30-836cbf3ce384-isvc-xgboost-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:35:48.237918 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.237926    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrwmq\" (UniqueName: \"kubernetes.io/projected/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kube-api-access-wrwmq\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:35:48.238103 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.237936    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71c8be97-9fba-4f2a-bb30-836cbf3ce384-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:35:48.827690 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.827655    2558 generic.go:356] "Generic (PLEG): container finished" podID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerID="5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630" exitCode=0
Jul 10 19:35:48.827878 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.827719    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"
Jul 10 19:35:48.827878 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.827723    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerDied","Data":"5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630"}
Jul 10 19:35:48.827878 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.827867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv" event={"ID":"71c8be97-9fba-4f2a-bb30-836cbf3ce384","Type":"ContainerDied","Data":"7c6bc9bfbdfe8b2b021c9939b8b4e99c3047004228aebfc350e61ca181a4ee27"}
Jul 10 19:35:48.827989 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.827893    2558 scope.go:117] "RemoveContainer" containerID="39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309"
Jul 10 19:35:48.836803 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.836779    2558 scope.go:117] "RemoveContainer" containerID="5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630"
Jul 10 19:35:48.843620 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.843603    2558 scope.go:117] "RemoveContainer" containerID="913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c"
Jul 10 19:35:48.851270 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.851242    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"]
Jul 10 19:35:48.855796 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.855774    2558 scope.go:117] "RemoveContainer" containerID="39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309"
Jul 10 19:35:48.855796 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.855787    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-6446b8b9dc-9t8gv"]
Jul 10 19:35:48.856037 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:35:48.856020    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309\": container with ID starting with 39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309 not found: ID does not exist" containerID="39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309"
Jul 10 19:35:48.856081 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.856045    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309"} err="failed to get container status \"39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309\": rpc error: code = NotFound desc = could not find container \"39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309\": container with ID starting with 39b0600310978a8e93af53d72a9a74d7017a1014197ee099975c02e696604309 not found: ID does not exist"
Jul 10 19:35:48.856081 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.856062    2558 scope.go:117] "RemoveContainer" containerID="5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630"
Jul 10 19:35:48.856303 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:35:48.856289    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630\": container with ID starting with 5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630 not found: ID does not exist" containerID="5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630"
Jul 10 19:35:48.856356 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.856306    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630"} err="failed to get container status \"5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630\": rpc error: code = NotFound desc = could not find container \"5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630\": container with ID starting with 5bbde080c38d1646a106646045e81ae46cd48efe33f611df674ee3a2b9fea630 not found: ID does not exist"
Jul 10 19:35:48.856356 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.856318    2558 scope.go:117] "RemoveContainer" containerID="913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c"
Jul 10 19:35:48.856553 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:35:48.856536    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c\": container with ID starting with 913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c not found: ID does not exist" containerID="913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c"
Jul 10 19:35:48.856589 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:48.856558    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c"} err="failed to get container status \"913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c\": rpc error: code = NotFound desc = could not find container \"913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c\": container with ID starting with 913f7c3b0e0ff612b3bd4be87f0d16f7b566c637bdc8ae85fa96003e86cc387c not found: ID does not exist"
Jul 10 19:35:49.256904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:49.256877    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" path="/var/lib/kubelet/pods/71c8be97-9fba-4f2a-bb30-836cbf3ce384/volumes"
Jul 10 19:35:49.834428 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:49.834397    2558 generic.go:356] "Generic (PLEG): container finished" podID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerID="c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e" exitCode=0
Jul 10 19:35:49.834428 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:49.834430    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerDied","Data":"c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e"}
Jul 10 19:35:50.840008 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:50.839974    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerStarted","Data":"f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb"}
Jul 10 19:35:50.840453 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:50.840036    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerStarted","Data":"29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9"}
Jul 10 19:35:50.840453 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:50.840225    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:50.860165 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:50.860102    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" podStartSLOduration=3.377575831 podStartE2EDuration="6.860083634s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.482507803s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:35:50.859582947 +0000 UTC m=+3396.121247731" watchObservedRunningTime="2026-07-10 19:35:50.860083634 +0000 UTC m=+3396.121748408"
Jul 10 19:35:51.844167 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:51.844137    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:35:57.852726 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:35:57.852694    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:36:27.902629 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:27.902586    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 10 19:36:37.855839 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:37.855809    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:36:44.676847 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.676813    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"]
Jul 10 19:36:44.677683 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.677554    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kube-rbac-proxy" containerID="cri-o://f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb" gracePeriod=30
Jul 10 19:36:44.682872 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.677403    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kserve-container" containerID="cri-o://29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9" gracePeriod=30
Jul 10 19:36:44.752483 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752450    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"]
Jul 10 19:36:44.752733 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752722    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container"
Jul 10 19:36:44.752815 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752735    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container"
Jul 10 19:36:44.752815 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752768    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="storage-initializer"
Jul 10 19:36:44.752815 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752773    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="storage-initializer"
Jul 10 19:36:44.752815 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752781    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kube-rbac-proxy"
Jul 10 19:36:44.752815 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752786    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kube-rbac-proxy"
Jul 10 19:36:44.752991 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752830    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kube-rbac-proxy"
Jul 10 19:36:44.752991 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.752838    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c8be97-9fba-4f2a-bb30-836cbf3ce384" containerName="kserve-container"
Jul 10 19:36:44.755287 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.755265    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.758076 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.758034    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 10 19:36:44.758198 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.758067    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 10 19:36:44.766061 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.766040    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"]
Jul 10 19:36:44.838015 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.837983    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.838152 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.838024    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftdt\" (UniqueName: \"kubernetes.io/projected/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kube-api-access-mftdt\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.838152 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.838046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.838152 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.838125    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.938829 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.938738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftdt\" (UniqueName: \"kubernetes.io/projected/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kube-api-access-mftdt\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.938829 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.938799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.938997 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.938861    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.938997 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.938907    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.938997 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:36:44.938959    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 10 19:36:44.939127 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:36:44.939038    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls podName:3f778bb4-64ff-4e65-b48b-c2dcd1b08d22 nodeName:}" failed. No retries permitted until 2026-07-10 19:36:45.439017761 +0000 UTC m=+3450.700682516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls") pod "isvc-xgboost-v2-predictor-5766b884c9-wqd6l" (UID: "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 10 19:36:44.939263 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.939247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.939554 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.939537    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:44.949303 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:44.949278    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftdt\" (UniqueName: \"kubernetes.io/projected/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kube-api-access-mftdt\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:45.026715 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:45.026683    2558 generic.go:356] "Generic (PLEG): container finished" podID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerID="f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb" exitCode=2
Jul 10 19:36:45.026886 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:45.026768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerDied","Data":"f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb"}
Jul 10 19:36:45.444332 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:45.444291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:45.446638 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:45.446616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls\") pod \"isvc-xgboost-v2-predictor-5766b884c9-wqd6l\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:45.666187 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:45.666149    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:45.792273 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:45.792205    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"]
Jul 10 19:36:45.794781 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:36:45.794737    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f778bb4_64ff_4e65_b48b_c2dcd1b08d22.slice/crio-a49b6af7eaf2074014ca31f657d9139c7cd85dd12e0e5cf9204bfbb1806a18b0 WatchSource:0}: Error finding container a49b6af7eaf2074014ca31f657d9139c7cd85dd12e0e5cf9204bfbb1806a18b0: Status 404 returned error can't find the container with id a49b6af7eaf2074014ca31f657d9139c7cd85dd12e0e5cf9204bfbb1806a18b0
Jul 10 19:36:46.032263 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:46.032230    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerStarted","Data":"98ca74005a0d6721f069c1a3affb19f9c6e09290bd29fd0abe5cd157f430f0e2"}
Jul 10 19:36:46.032436 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:46.032290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerStarted","Data":"a49b6af7eaf2074014ca31f657d9139c7cd85dd12e0e5cf9204bfbb1806a18b0"}
Jul 10 19:36:47.848541 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:47.848494    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Jul 10 19:36:50.048872 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:50.048841    2558 generic.go:356] "Generic (PLEG): container finished" podID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerID="98ca74005a0d6721f069c1a3affb19f9c6e09290bd29fd0abe5cd157f430f0e2" exitCode=0
Jul 10 19:36:50.049276 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:50.048889    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerDied","Data":"98ca74005a0d6721f069c1a3affb19f9c6e09290bd29fd0abe5cd157f430f0e2"}
Jul 10 19:36:51.055269 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.055235    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerStarted","Data":"fdf22fb806ed928f15baff8dc3dd9c79c0108296b686fb53f470471bb0f9d1c1"}
Jul 10 19:36:51.055642 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.055296    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerStarted","Data":"9fe7eebbd0b86bccaef157b74bb69a9a1cc619a318b3e79225ef7f8b41a9ae6f"}
Jul 10 19:36:51.055642 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.055476    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:51.074665 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.074615    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podStartSLOduration=3.288927399 podStartE2EDuration="7.074599907s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785672508s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:36:51.073164236 +0000 UTC m=+3456.334829010" watchObservedRunningTime="2026-07-10 19:36:51.074599907 +0000 UTC m=+3456.336264679"
Jul 10 19:36:51.920967 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.920944    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:36:51.990508 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.990436    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") "
Jul 10 19:36:51.990508 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.990469    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls\") pod \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") "
Jul 10 19:36:51.990508 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.990490    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kserve-provision-location\") pod \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") "
Jul 10 19:36:51.990702 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.990546    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgwlp\" (UniqueName: \"kubernetes.io/projected/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kube-api-access-rgwlp\") pod \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\" (UID: \"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93\") "
Jul 10 19:36:51.990901 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.990874    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" (UID: "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:36:51.990973 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.990940    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" (UID: "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:36:51.992559 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.992538    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" (UID: "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:36:51.992669 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:51.992574    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kube-api-access-rgwlp" (OuterVolumeSpecName: "kube-api-access-rgwlp") pod "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" (UID: "c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93"). InnerVolumeSpecName "kube-api-access-rgwlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:36:52.060183 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.060155    2558 generic.go:356] "Generic (PLEG): container finished" podID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerID="29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9" exitCode=0
Jul 10 19:36:52.060546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.060206    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"
Jul 10 19:36:52.060546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.060243    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerDied","Data":"29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9"}
Jul 10 19:36:52.060546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.060302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d" event={"ID":"c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93","Type":"ContainerDied","Data":"417208b4a4243e412c50d31c4e6cf5a527c02bcd723047dd4c8149b45acc6771"}
Jul 10 19:36:52.060546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.060328    2558 scope.go:117] "RemoveContainer" containerID="f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb"
Jul 10 19:36:52.060796 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.060776    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:52.062276 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.062252    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:36:52.068595 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.068578    2558 scope.go:117] "RemoveContainer" containerID="29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9"
Jul 10 19:36:52.075693 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.075678    2558 scope.go:117] "RemoveContainer" containerID="c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e"
Jul 10 19:36:52.081059 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.081039    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"]
Jul 10 19:36:52.083076 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.083065    2558 scope.go:117] "RemoveContainer" containerID="f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb"
Jul 10 19:36:52.083317 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:36:52.083298    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb\": container with ID starting with f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb not found: ID does not exist" containerID="f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb"
Jul 10 19:36:52.083385 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.083331    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb"} err="failed to get container status \"f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb\": rpc error: code = NotFound desc = could not find container \"f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb\": container with ID starting with f6e1e0ffb06d987e3411f836d8401a3ec274617f7d515c8a5e456686f750a2cb not found: ID does not exist"
Jul 10 19:36:52.083385 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.083354    2558 scope.go:117] "RemoveContainer" containerID="29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9"
Jul 10 19:36:52.083591 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:36:52.083575    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9\": container with ID starting with 29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9 not found: ID does not exist" containerID="29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9"
Jul 10 19:36:52.083630 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.083597    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9"} err="failed to get container status \"29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9\": rpc error: code = NotFound desc = could not find container \"29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9\": container with ID starting with 29f48cb1ddcde329a7a9c0c9aabd43b16826d6fe99db9484b4275102381e9ae9 not found: ID does not exist"
Jul 10 19:36:52.083630 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.083613    2558 scope.go:117] "RemoveContainer" containerID="c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e"
Jul 10 19:36:52.083869 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:36:52.083853    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e\": container with ID starting with c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e not found: ID does not exist" containerID="c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e"
Jul 10 19:36:52.083919 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.083874    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e"} err="failed to get container status \"c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e\": rpc error: code = NotFound desc = could not find container \"c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e\": container with ID starting with c027e9de645544436d15503932e06746f94c41b4f8881ce225b984767ce2d11e not found: ID does not exist"
Jul 10 19:36:52.086450 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.086429    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-79cd9b44fc-v2r8d"]
Jul 10 19:36:52.091471 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.091453    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgwlp\" (UniqueName: \"kubernetes.io/projected/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kube-api-access-rgwlp\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:36:52.091553 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.091477    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:36:52.091553 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.091494    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:36:52.091553 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:52.091507    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:36:53.065411 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:53.065377    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:36:53.256801 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:53.256769    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" path="/var/lib/kubelet/pods/c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93/volumes"
Jul 10 19:36:58.070364 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:58.070331    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:36:58.070868 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:36:58.070835    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:37:08.071319 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:08.071278    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:37:18.071284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:18.071240    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:37:28.071284 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:28.071244    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:37:38.071440 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:38.071354    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:37:48.071946 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:48.071915    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:37:54.853445 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.853413    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"]
Jul 10 19:37:54.853858 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.853639    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" containerID="cri-o://9fe7eebbd0b86bccaef157b74bb69a9a1cc619a318b3e79225ef7f8b41a9ae6f" gracePeriod=30
Jul 10 19:37:54.853858 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.853670    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kube-rbac-proxy" containerID="cri-o://fdf22fb806ed928f15baff8dc3dd9c79c0108296b686fb53f470471bb0f9d1c1" gracePeriod=30
Jul 10 19:37:54.938509 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938477    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"]
Jul 10 19:37:54.938770 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938758    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="storage-initializer"
Jul 10 19:37:54.938826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938772    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="storage-initializer"
Jul 10 19:37:54.938826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938785    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kube-rbac-proxy"
Jul 10 19:37:54.938826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938790    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kube-rbac-proxy"
Jul 10 19:37:54.938826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938796    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kserve-container"
Jul 10 19:37:54.938826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938802    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kserve-container"
Jul 10 19:37:54.938977 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938850    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kserve-container"
Jul 10 19:37:54.938977 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.938862    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c32fe8bb-2cdf-4738-b8b5-c9a8f2444b93" containerName="kube-rbac-proxy"
Jul 10 19:37:54.941793 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.941774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:54.943998 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.943971    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jul 10 19:37:54.944097 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.944078    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jul 10 19:37:54.944179 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.944078    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 10 19:37:54.952851 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:54.952830    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"]
Jul 10 19:37:55.010746 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.010722    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2455825f-0519-4a5c-8343-5637af266025-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.010855 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.010766    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.010855 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.010803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7fj\" (UniqueName: \"kubernetes.io/projected/2455825f-0519-4a5c-8343-5637af266025-kube-api-access-5c7fj\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.010958 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.010872    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2455825f-0519-4a5c-8343-5637af266025-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.111247 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.111177    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2455825f-0519-4a5c-8343-5637af266025-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.111247 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.111228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2455825f-0519-4a5c-8343-5637af266025-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.111426 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.111248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.111426 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.111282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7fj\" (UniqueName: \"kubernetes.io/projected/2455825f-0519-4a5c-8343-5637af266025-kube-api-access-5c7fj\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.111540 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:37:55.111431    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 10 19:37:55.111540 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:37:55.111507    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls podName:2455825f-0519-4a5c-8343-5637af266025 nodeName:}" failed. No retries permitted until 2026-07-10 19:37:55.611482294 +0000 UTC m=+3520.873147046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls") pod "isvc-sklearn-s3-predictor-84b6676767-76kvb" (UID: "2455825f-0519-4a5c-8343-5637af266025") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jul 10 19:37:55.111617 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.111589    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2455825f-0519-4a5c-8343-5637af266025-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.111957 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.111939    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2455825f-0519-4a5c-8343-5637af266025-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.119505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.119486    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7fj\" (UniqueName: \"kubernetes.io/projected/2455825f-0519-4a5c-8343-5637af266025-kube-api-access-5c7fj\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.282529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.282499    2558 generic.go:356] "Generic (PLEG): container finished" podID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerID="fdf22fb806ed928f15baff8dc3dd9c79c0108296b686fb53f470471bb0f9d1c1" exitCode=2
Jul 10 19:37:55.282676 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.282570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerDied","Data":"fdf22fb806ed928f15baff8dc3dd9c79c0108296b686fb53f470471bb0f9d1c1"}
Jul 10 19:37:55.615452 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.615423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.617710 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.617689    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84b6676767-76kvb\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.852081 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.852052    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:55.979886 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:55.979865    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"]
Jul 10 19:37:55.981726 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:37:55.981691    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2455825f_0519_4a5c_8343_5637af266025.slice/crio-ce107d2839996a677a367f5b2e762fe72a199eaebaeb6a57fbb790660d1f32d7 WatchSource:0}: Error finding container ce107d2839996a677a367f5b2e762fe72a199eaebaeb6a57fbb790660d1f32d7: Status 404 returned error can't find the container with id ce107d2839996a677a367f5b2e762fe72a199eaebaeb6a57fbb790660d1f32d7
Jul 10 19:37:56.288379 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:56.288345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerStarted","Data":"e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708"}
Jul 10 19:37:56.288541 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:56.288398    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerStarted","Data":"ce107d2839996a677a367f5b2e762fe72a199eaebaeb6a57fbb790660d1f32d7"}
Jul 10 19:37:57.293150 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:57.293116    2558 generic.go:356] "Generic (PLEG): container finished" podID="2455825f-0519-4a5c-8343-5637af266025" containerID="e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708" exitCode=0
Jul 10 19:37:57.293552 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:57.293165    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerDied","Data":"e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708"}
Jul 10 19:37:58.065854 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.065817    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Jul 10 19:37:58.071214 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.071187    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Jul 10 19:37:58.299003 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.298965    2558 generic.go:356] "Generic (PLEG): container finished" podID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerID="9fe7eebbd0b86bccaef157b74bb69a9a1cc619a318b3e79225ef7f8b41a9ae6f" exitCode=0
Jul 10 19:37:58.299003 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.298993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerDied","Data":"9fe7eebbd0b86bccaef157b74bb69a9a1cc619a318b3e79225ef7f8b41a9ae6f"}
Jul 10 19:37:58.301588 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.301568    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerStarted","Data":"93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79"}
Jul 10 19:37:58.301701 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.301611    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerStarted","Data":"befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3"}
Jul 10 19:37:58.301701 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.301692    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:58.321385 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.321285    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podStartSLOduration=3.412918629 podStartE2EDuration="4.321265561s" totalImagesPullingTime="0s" totalInitContainerRuntime="908.346932ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:37:58.319511534 +0000 UTC m=+3523.581176308" watchObservedRunningTime="2026-07-10 19:37:58.321265561 +0000 UTC m=+3523.582930335"
Jul 10 19:37:58.401091 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.401065    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:37:58.433756 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.433714    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls\") pod \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") "
Jul 10 19:37:58.433908 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.433766    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mftdt\" (UniqueName: \"kubernetes.io/projected/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kube-api-access-mftdt\") pod \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") "
Jul 10 19:37:58.433908 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.433786    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kserve-provision-location\") pod \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") "
Jul 10 19:37:58.433908 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.433844    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") pod \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\" (UID: \"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22\") "
Jul 10 19:37:58.434147 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.434123    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" (UID: "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:37:58.434216 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.434187    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-isvc-xgboost-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-kube-rbac-proxy-sar-config") pod "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" (UID: "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22"). InnerVolumeSpecName "isvc-xgboost-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:37:58.435711 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.435686    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" (UID: "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:37:58.435855 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.435817    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kube-api-access-mftdt" (OuterVolumeSpecName: "kube-api-access-mftdt") pod "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" (UID: "3f778bb4-64ff-4e65-b48b-c2dcd1b08d22"). InnerVolumeSpecName "kube-api-access-mftdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:37:58.534572 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.534549    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:37:58.534572 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.534568    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftdt\" (UniqueName: \"kubernetes.io/projected/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kube-api-access-mftdt\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:37:58.534719 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.534578    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:37:58.534719 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:58.534589    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22-isvc-xgboost-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:37:59.307128 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.307096    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"
Jul 10 19:37:59.307592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.307126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l" event={"ID":"3f778bb4-64ff-4e65-b48b-c2dcd1b08d22","Type":"ContainerDied","Data":"a49b6af7eaf2074014ca31f657d9139c7cd85dd12e0e5cf9204bfbb1806a18b0"}
Jul 10 19:37:59.307592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.307191    2558 scope.go:117] "RemoveContainer" containerID="fdf22fb806ed928f15baff8dc3dd9c79c0108296b686fb53f470471bb0f9d1c1"
Jul 10 19:37:59.307794 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.307732    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:37:59.309194 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.309059    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:37:59.315395 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.315377    2558 scope.go:117] "RemoveContainer" containerID="9fe7eebbd0b86bccaef157b74bb69a9a1cc619a318b3e79225ef7f8b41a9ae6f"
Jul 10 19:37:59.322242 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.322229    2558 scope.go:117] "RemoveContainer" containerID="98ca74005a0d6721f069c1a3affb19f9c6e09290bd29fd0abe5cd157f430f0e2"
Jul 10 19:37:59.324689 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.324669    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"]
Jul 10 19:37:59.328343 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:37:59.328323    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-5766b884c9-wqd6l"]
Jul 10 19:38:00.311579 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:00.311538    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:38:01.256519 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:01.256484    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" path="/var/lib/kubelet/pods/3f778bb4-64ff-4e65-b48b-c2dcd1b08d22/volumes"
Jul 10 19:38:05.316894 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:05.316867    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:38:05.317393 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:05.317364    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:38:15.317480 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:15.317445    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:38:25.317823 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:25.317786    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:38:35.317876 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:35.317830    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:38:45.317382 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:45.317343    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:38:55.317846 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:38:55.317799    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:39:05.318648 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:05.318617    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:39:15.041805 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.041763    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"]
Jul 10 19:39:15.042256 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.042054    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" containerID="cri-o://befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3" gracePeriod=30
Jul 10 19:39:15.042256 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.042089    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kube-rbac-proxy" containerID="cri-o://93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79" gracePeriod=30
Jul 10 19:39:15.156525 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156493    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"]
Jul 10 19:39:15.156826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156792    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container"
Jul 10 19:39:15.156826 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156808    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container"
Jul 10 19:39:15.156987 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156845    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="storage-initializer"
Jul 10 19:39:15.156987 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156855    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="storage-initializer"
Jul 10 19:39:15.156987 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156865    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kube-rbac-proxy"
Jul 10 19:39:15.156987 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156871    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kube-rbac-proxy"
Jul 10 19:39:15.156987 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156921    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kube-rbac-proxy"
Jul 10 19:39:15.156987 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.156929    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f778bb4-64ff-4e65-b48b-c2dcd1b08d22" containerName="kserve-container"
Jul 10 19:39:15.160242 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.160227    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.162392 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.162375    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 10 19:39:15.162487 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.162444    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:39:15.162487 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.162380    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 10 19:39:15.169767 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.169725    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"]
Jul 10 19:39:15.262286 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.262253    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.262460 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.262331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.262460 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.262370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqgc\" (UniqueName: \"kubernetes.io/projected/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kube-api-access-hjqgc\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.262580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.262462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.262580 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.262527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.312545 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.312457    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Jul 10 19:39:15.317294 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.317263    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Jul 10 19:39:15.363185 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.363158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.363306 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.363203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqgc\" (UniqueName: \"kubernetes.io/projected/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kube-api-access-hjqgc\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.363401 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.363370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.363475 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.363458    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.363565 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.363531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.363706 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.363585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.366107 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.366090    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 10 19:39:15.366182 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.366147    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 10 19:39:15.366258 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.366245    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:39:15.371703 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.371682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqgc\" (UniqueName: \"kubernetes.io/projected/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kube-api-access-hjqgc\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.374951 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.374932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.375055 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.374970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.376770 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.376730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.457550 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.457528    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:39:15.471557 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.471532    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:15.578934 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.578855    2558 generic.go:356] "Generic (PLEG): container finished" podID="2455825f-0519-4a5c-8343-5637af266025" containerID="93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79" exitCode=2
Jul 10 19:39:15.579068 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.578921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerDied","Data":"93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79"}
Jul 10 19:39:15.603329 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:15.603303    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"]
Jul 10 19:39:15.605836 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:39:15.605789    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e00ea4e_4ab4_4e03_9ea8_5857c948597c.slice/crio-6aa6a77923bb8404d19a0ac0cee7645162cf5d6226cece0cb94b0ce7445036a7 WatchSource:0}: Error finding container 6aa6a77923bb8404d19a0ac0cee7645162cf5d6226cece0cb94b0ce7445036a7: Status 404 returned error can't find the container with id 6aa6a77923bb8404d19a0ac0cee7645162cf5d6226cece0cb94b0ce7445036a7
Jul 10 19:39:16.584547 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:16.584520    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerStarted","Data":"45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7"}
Jul 10 19:39:16.584978 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:16.584584    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerStarted","Data":"6aa6a77923bb8404d19a0ac0cee7645162cf5d6226cece0cb94b0ce7445036a7"}
Jul 10 19:39:17.589980 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:17.589947    2558 generic.go:356] "Generic (PLEG): container finished" podID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerID="45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7" exitCode=0
Jul 10 19:39:17.590346 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:17.590034    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerDied","Data":"45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7"}
Jul 10 19:39:18.595245 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:18.595210    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerStarted","Data":"1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1"}
Jul 10 19:39:18.595676 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:18.595268    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerStarted","Data":"e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d"}
Jul 10 19:39:18.595676 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:18.595295    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:18.595676 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:18.595328    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:18.596411 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:18.596386    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:39:18.615481 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:18.615380    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podStartSLOduration=2.694743685 podStartE2EDuration="3.615359855s" totalImagesPullingTime="0s" totalInitContainerRuntime="920.61617ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:39:18.61424108 +0000 UTC m=+3603.875905866" watchObservedRunningTime="2026-07-10 19:39:18.615359855 +0000 UTC m=+3603.877024631"
Jul 10 19:39:18.700719 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:18.700700    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:39:19.187878 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.187853    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:39:19.295698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.295631    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2455825f-0519-4a5c-8343-5637af266025-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"2455825f-0519-4a5c-8343-5637af266025\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") "
Jul 10 19:39:19.295698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.295666    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c7fj\" (UniqueName: \"kubernetes.io/projected/2455825f-0519-4a5c-8343-5637af266025-kube-api-access-5c7fj\") pod \"2455825f-0519-4a5c-8343-5637af266025\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") "
Jul 10 19:39:19.295698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.295686    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls\") pod \"2455825f-0519-4a5c-8343-5637af266025\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") "
Jul 10 19:39:19.295994 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.295755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2455825f-0519-4a5c-8343-5637af266025-kserve-provision-location\") pod \"2455825f-0519-4a5c-8343-5637af266025\" (UID: \"2455825f-0519-4a5c-8343-5637af266025\") "
Jul 10 19:39:19.296058 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.296016    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2455825f-0519-4a5c-8343-5637af266025-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "2455825f-0519-4a5c-8343-5637af266025" (UID: "2455825f-0519-4a5c-8343-5637af266025"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:39:19.296133 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.296112    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2455825f-0519-4a5c-8343-5637af266025-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2455825f-0519-4a5c-8343-5637af266025" (UID: "2455825f-0519-4a5c-8343-5637af266025"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:39:19.297655 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.297632    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2455825f-0519-4a5c-8343-5637af266025-kube-api-access-5c7fj" (OuterVolumeSpecName: "kube-api-access-5c7fj") pod "2455825f-0519-4a5c-8343-5637af266025" (UID: "2455825f-0519-4a5c-8343-5637af266025"). InnerVolumeSpecName "kube-api-access-5c7fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:39:19.297780 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.297707    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2455825f-0519-4a5c-8343-5637af266025" (UID: "2455825f-0519-4a5c-8343-5637af266025"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:39:19.396418 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.396394    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2455825f-0519-4a5c-8343-5637af266025-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:39:19.396418 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.396417    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c7fj\" (UniqueName: \"kubernetes.io/projected/2455825f-0519-4a5c-8343-5637af266025-kube-api-access-5c7fj\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:39:19.396573 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.396427    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2455825f-0519-4a5c-8343-5637af266025-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:39:19.396573 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.396435    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2455825f-0519-4a5c-8343-5637af266025-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:39:19.600374 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.600298    2558 generic.go:356] "Generic (PLEG): container finished" podID="2455825f-0519-4a5c-8343-5637af266025" containerID="befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3" exitCode=0
Jul 10 19:39:19.600374 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.600358    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"
Jul 10 19:39:19.600787 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.600379    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerDied","Data":"befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3"}
Jul 10 19:39:19.600787 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.600442    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb" event={"ID":"2455825f-0519-4a5c-8343-5637af266025","Type":"ContainerDied","Data":"ce107d2839996a677a367f5b2e762fe72a199eaebaeb6a57fbb790660d1f32d7"}
Jul 10 19:39:19.600787 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.600470    2558 scope.go:117] "RemoveContainer" containerID="93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79"
Jul 10 19:39:19.601208 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.601161    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:39:19.615062 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.615045    2558 scope.go:117] "RemoveContainer" containerID="befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3"
Jul 10 19:39:19.622009 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.621992    2558 scope.go:117] "RemoveContainer" containerID="e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708"
Jul 10 19:39:19.625423 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.625404    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"]
Jul 10 19:39:19.629255 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.629236    2558 scope.go:117] "RemoveContainer" containerID="93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79"
Jul 10 19:39:19.629521 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:39:19.629505    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79\": container with ID starting with 93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79 not found: ID does not exist" containerID="93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79"
Jul 10 19:39:19.629579 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.629529    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79"} err="failed to get container status \"93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79\": rpc error: code = NotFound desc = could not find container \"93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79\": container with ID starting with 93b780907b0b51ad2f290ce3d473ff0bb830fbbc7c5f2d015ffb660fa98bcd79 not found: ID does not exist"
Jul 10 19:39:19.629579 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.629547    2558 scope.go:117] "RemoveContainer" containerID="befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3"
Jul 10 19:39:19.629779 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:39:19.629764    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3\": container with ID starting with befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3 not found: ID does not exist" containerID="befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3"
Jul 10 19:39:19.629822 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.629784    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3"} err="failed to get container status \"befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3\": rpc error: code = NotFound desc = could not find container \"befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3\": container with ID starting with befcc8eee4bfbcdff95e5bae24b1e9f84e29086bec6a10ed556572d0853b8ef3 not found: ID does not exist"
Jul 10 19:39:19.629822 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.629796    2558 scope.go:117] "RemoveContainer" containerID="e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708"
Jul 10 19:39:19.630008 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:39:19.629994    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708\": container with ID starting with e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708 not found: ID does not exist" containerID="e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708"
Jul 10 19:39:19.630083 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.630012    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708"} err="failed to get container status \"e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708\": rpc error: code = NotFound desc = could not find container \"e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708\": container with ID starting with e264b1bc6825cc9f38c25b3c5b20292a3f5884d928ded01d6f2f4eda95a13708 not found: ID does not exist"
Jul 10 19:39:19.632619 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:19.632600    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84b6676767-76kvb"]
Jul 10 19:39:21.257579 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:21.257547    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2455825f-0519-4a5c-8343-5637af266025" path="/var/lib/kubelet/pods/2455825f-0519-4a5c-8343-5637af266025/volumes"
Jul 10 19:39:24.604677 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:24.604651    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:39:24.605147 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:24.605109    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:39:34.605227 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:34.605180    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:39:44.605149 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:44.605110    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:39:54.605673 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:39:54.605634    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:40:04.605365 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:04.605314    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:40:14.605191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:14.605148    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Jul 10 19:40:24.605877 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:24.605851    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:40:25.197657 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:25.197627    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"]
Jul 10 19:40:25.197900 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:25.197844    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container" containerID="cri-o://e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d" gracePeriod=30
Jul 10 19:40:25.197975 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:25.197871    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kube-rbac-proxy" containerID="cri-o://1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1" gracePeriod=30
Jul 10 19:40:25.833246 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:25.833211    2558 generic.go:356] "Generic (PLEG): container finished" podID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerID="1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1" exitCode=2
Jul 10 19:40:25.833642 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:25.833290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerDied","Data":"1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1"}
Jul 10 19:40:26.274763 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.274708    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"]
Jul 10 19:40:26.275073 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275055    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="storage-initializer"
Jul 10 19:40:26.275161 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275075    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="storage-initializer"
Jul 10 19:40:26.275161 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275089    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kube-rbac-proxy"
Jul 10 19:40:26.275161 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275097    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kube-rbac-proxy"
Jul 10 19:40:26.275161 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275127    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container"
Jul 10 19:40:26.275161 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275136    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container"
Jul 10 19:40:26.275415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275217    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kube-rbac-proxy"
Jul 10 19:40:26.275415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.275230    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2455825f-0519-4a5c-8343-5637af266025" containerName="kserve-container"
Jul 10 19:40:26.278033 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.278012    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.280172 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.280152    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jul 10 19:40:26.280285 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.280181    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jul 10 19:40:26.289468 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.289437    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"]
Jul 10 19:40:26.359988 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.359955    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.360131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.359998    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wtf\" (UniqueName: \"kubernetes.io/projected/772265d4-cc7b-4d1b-bd74-a158b2adf520-kube-api-access-79wtf\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.360131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.360031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772265d4-cc7b-4d1b-bd74-a158b2adf520-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.360131 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.360076    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772265d4-cc7b-4d1b-bd74-a158b2adf520-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.460668 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.460640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772265d4-cc7b-4d1b-bd74-a158b2adf520-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.460811 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.460691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.460811 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.460714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wtf\" (UniqueName: \"kubernetes.io/projected/772265d4-cc7b-4d1b-bd74-a158b2adf520-kube-api-access-79wtf\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.460811 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.460764    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772265d4-cc7b-4d1b-bd74-a158b2adf520-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.461003 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:26.460849    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 10 19:40:26.461003 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:26.460924    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls podName:772265d4-cc7b-4d1b-bd74-a158b2adf520 nodeName:}" failed. No retries permitted until 2026-07-10 19:40:26.960906703 +0000 UTC m=+3672.222571457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" (UID: "772265d4-cc7b-4d1b-bd74-a158b2adf520") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
Jul 10 19:40:26.461128 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.461104    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772265d4-cc7b-4d1b-bd74-a158b2adf520-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.461413 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.461394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772265d4-cc7b-4d1b-bd74-a158b2adf520-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.469311 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.469295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wtf\" (UniqueName: \"kubernetes.io/projected/772265d4-cc7b-4d1b-bd74-a158b2adf520-kube-api-access-79wtf\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.964624 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.964575    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:26.967033 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:26.967012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:27.187884 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:27.187848    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:27.321994 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:27.321962    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"]
Jul 10 19:40:27.324794 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:40:27.324762    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772265d4_cc7b_4d1b_bd74_a158b2adf520.slice/crio-680c9e71d9cc761b1345dcd17df23e64891ffe17543032a2d38b2921f5b3b9ba WatchSource:0}: Error finding container 680c9e71d9cc761b1345dcd17df23e64891ffe17543032a2d38b2921f5b3b9ba: Status 404 returned error can't find the container with id 680c9e71d9cc761b1345dcd17df23e64891ffe17543032a2d38b2921f5b3b9ba
Jul 10 19:40:27.847050 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:27.847009    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" event={"ID":"772265d4-cc7b-4d1b-bd74-a158b2adf520","Type":"ContainerStarted","Data":"1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7"}
Jul 10 19:40:27.847232 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:27.847128    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" event={"ID":"772265d4-cc7b-4d1b-bd74-a158b2adf520","Type":"ContainerStarted","Data":"680c9e71d9cc761b1345dcd17df23e64891ffe17543032a2d38b2921f5b3b9ba"}
Jul 10 19:40:29.135652 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:29.135614    2558 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e00ea4e_4ab4_4e03_9ea8_5857c948597c.slice/crio-conmon-e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d.scope\": RecentStats: unable to find data in memory cache]"
Jul 10 19:40:29.334114 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.334093    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:40:29.383157 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383088    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-cabundle-cert\") pod \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") "
Jul 10 19:40:29.383157 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383121    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjqgc\" (UniqueName: \"kubernetes.io/projected/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kube-api-access-hjqgc\") pod \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") "
Jul 10 19:40:29.383157 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383143    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kserve-provision-location\") pod \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") "
Jul 10 19:40:29.383356 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383161    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-proxy-tls\") pod \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") "
Jul 10 19:40:29.383356 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383284    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\" (UID: \"5e00ea4e-4ab4-4e03-9ea8-5857c948597c\") "
Jul 10 19:40:29.383482 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383438    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e00ea4e-4ab4-4e03-9ea8-5857c948597c" (UID: "5e00ea4e-4ab4-4e03-9ea8-5857c948597c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:40:29.383674 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383645    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5e00ea4e-4ab4-4e03-9ea8-5857c948597c" (UID: "5e00ea4e-4ab4-4e03-9ea8-5857c948597c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:40:29.383834 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.383689    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "5e00ea4e-4ab4-4e03-9ea8-5857c948597c" (UID: "5e00ea4e-4ab4-4e03-9ea8-5857c948597c"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:40:29.385195 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.385177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5e00ea4e-4ab4-4e03-9ea8-5857c948597c" (UID: "5e00ea4e-4ab4-4e03-9ea8-5857c948597c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:40:29.385289 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.385180    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kube-api-access-hjqgc" (OuterVolumeSpecName: "kube-api-access-hjqgc") pod "5e00ea4e-4ab4-4e03-9ea8-5857c948597c" (UID: "5e00ea4e-4ab4-4e03-9ea8-5857c948597c"). InnerVolumeSpecName "kube-api-access-hjqgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:40:29.484638 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.484613    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:29.484638 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.484636    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-cabundle-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:29.484827 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.484646    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjqgc\" (UniqueName: \"kubernetes.io/projected/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kube-api-access-hjqgc\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:29.484827 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.484657    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:29.484827 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.484667    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5e00ea4e-4ab4-4e03-9ea8-5857c948597c-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:29.857315 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.855877    2558 generic.go:356] "Generic (PLEG): container finished" podID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerID="e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d" exitCode=0
Jul 10 19:40:29.857315 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.855951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerDied","Data":"e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d"}
Jul 10 19:40:29.857315 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.856006    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p" event={"ID":"5e00ea4e-4ab4-4e03-9ea8-5857c948597c","Type":"ContainerDied","Data":"6aa6a77923bb8404d19a0ac0cee7645162cf5d6226cece0cb94b0ce7445036a7"}
Jul 10 19:40:29.857315 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.856043    2558 scope.go:117] "RemoveContainer" containerID="1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1"
Jul 10 19:40:29.857315 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.856207    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"
Jul 10 19:40:29.865803 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.865787    2558 scope.go:117] "RemoveContainer" containerID="e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d"
Jul 10 19:40:29.872752 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.872718    2558 scope.go:117] "RemoveContainer" containerID="45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7"
Jul 10 19:40:29.877564 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.877541    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"]
Jul 10 19:40:29.880214 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.880200    2558 scope.go:117] "RemoveContainer" containerID="1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1"
Jul 10 19:40:29.880466 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:29.880449    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1\": container with ID starting with 1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1 not found: ID does not exist" containerID="1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1"
Jul 10 19:40:29.880505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.880477    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1"} err="failed to get container status \"1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1\": rpc error: code = NotFound desc = could not find container \"1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1\": container with ID starting with 1eb57eff1ccd2bafff036afb320c4adefe5276ce6cdd4d9cd4e5d8eea14f03d1 not found: ID does not exist"
Jul 10 19:40:29.880505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.880495    2558 scope.go:117] "RemoveContainer" containerID="e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d"
Jul 10 19:40:29.880726 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:29.880712    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d\": container with ID starting with e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d not found: ID does not exist" containerID="e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d"
Jul 10 19:40:29.880786 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.880731    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d"} err="failed to get container status \"e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d\": rpc error: code = NotFound desc = could not find container \"e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d\": container with ID starting with e7f28176efd91f23b951af8f0f520d617a9783404b37b3a3b67696f840c86e1d not found: ID does not exist"
Jul 10 19:40:29.880786 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.880759    2558 scope.go:117] "RemoveContainer" containerID="45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7"
Jul 10 19:40:29.881019 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:29.881004    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7\": container with ID starting with 45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7 not found: ID does not exist" containerID="45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7"
Jul 10 19:40:29.881066 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.881024    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7"} err="failed to get container status \"45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7\": rpc error: code = NotFound desc = could not find container \"45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7\": container with ID starting with 45ce1138e506737fb26e51e068bffabee7ffab860a7c60735638420cef3a55a7 not found: ID does not exist"
Jul 10 19:40:29.883137 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:29.883117    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-65d66ff44c-vdm7p"]
Jul 10 19:40:31.256888 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:31.256858    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" path="/var/lib/kubelet/pods/5e00ea4e-4ab4-4e03-9ea8-5857c948597c/volumes"
Jul 10 19:40:31.865858 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:31.865835    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_772265d4-cc7b-4d1b-bd74-a158b2adf520/storage-initializer/0.log"
Jul 10 19:40:31.865994 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:31.865873    2558 generic.go:356] "Generic (PLEG): container finished" podID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerID="1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7" exitCode=1
Jul 10 19:40:31.865994 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:31.865954    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" event={"ID":"772265d4-cc7b-4d1b-bd74-a158b2adf520","Type":"ContainerDied","Data":"1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7"}
Jul 10 19:40:32.871603 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:32.871579    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_772265d4-cc7b-4d1b-bd74-a158b2adf520/storage-initializer/0.log"
Jul 10 19:40:32.871980 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:32.871636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" event={"ID":"772265d4-cc7b-4d1b-bd74-a158b2adf520","Type":"ContainerStarted","Data":"c59a2cbec80ce617383a8067f07c96796171b55808d8abeb8cb824453d3c7084"}
Jul 10 19:40:35.884601 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:35.884571    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_772265d4-cc7b-4d1b-bd74-a158b2adf520/storage-initializer/1.log"
Jul 10 19:40:35.885005 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:35.884973    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_772265d4-cc7b-4d1b-bd74-a158b2adf520/storage-initializer/0.log"
Jul 10 19:40:35.885052 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:35.885005    2558 generic.go:356] "Generic (PLEG): container finished" podID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerID="c59a2cbec80ce617383a8067f07c96796171b55808d8abeb8cb824453d3c7084" exitCode=1
Jul 10 19:40:35.885104 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:35.885083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" event={"ID":"772265d4-cc7b-4d1b-bd74-a158b2adf520","Type":"ContainerDied","Data":"c59a2cbec80ce617383a8067f07c96796171b55808d8abeb8cb824453d3c7084"}
Jul 10 19:40:35.885155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:35.885149    2558 scope.go:117] "RemoveContainer" containerID="1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7"
Jul 10 19:40:35.885505 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:35.885482    2558 scope.go:117] "RemoveContainer" containerID="1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7"
Jul 10 19:40:35.895480 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:35.895461    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_kserve-ci-e2e-test(772265d4-cc7b-4d1b-bd74-a158b2adf520)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520"
Jul 10 19:40:35.895609 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:40:35.895585    2558 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_kserve-ci-e2e-test_772265d4-cc7b-4d1b-bd74-a158b2adf520_0 in pod sandbox 680c9e71d9cc761b1345dcd17df23e64891ffe17543032a2d38b2921f5b3b9ba from index: no such id: '1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7'" containerID="1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7"
Jul 10 19:40:35.895658 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:35.895615    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_kserve-ci-e2e-test_772265d4-cc7b-4d1b-bd74-a158b2adf520_0 in pod sandbox 680c9e71d9cc761b1345dcd17df23e64891ffe17543032a2d38b2921f5b3b9ba from index: no such id: '1d07cefa9cb6fd380d20f2ce7b540548dfc411ea98641befedfbeda33bc735a7'"
Jul 10 19:40:36.278556 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:36.278526    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"]
Jul 10 19:40:36.889562 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:36.889537    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_772265d4-cc7b-4d1b-bd74-a158b2adf520/storage-initializer/1.log"
Jul 10 19:40:37.014919 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.014896    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_772265d4-cc7b-4d1b-bd74-a158b2adf520/storage-initializer/1.log"
Jul 10 19:40:37.018134 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.018120    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:37.033418 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.033398    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772265d4-cc7b-4d1b-bd74-a158b2adf520-kserve-provision-location\") pod \"772265d4-cc7b-4d1b-bd74-a158b2adf520\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") "
Jul 10 19:40:37.033482 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.033439    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772265d4-cc7b-4d1b-bd74-a158b2adf520-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"772265d4-cc7b-4d1b-bd74-a158b2adf520\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") "
Jul 10 19:40:37.033529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.033485    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79wtf\" (UniqueName: \"kubernetes.io/projected/772265d4-cc7b-4d1b-bd74-a158b2adf520-kube-api-access-79wtf\") pod \"772265d4-cc7b-4d1b-bd74-a158b2adf520\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") "
Jul 10 19:40:37.033529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.033509    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls\") pod \"772265d4-cc7b-4d1b-bd74-a158b2adf520\" (UID: \"772265d4-cc7b-4d1b-bd74-a158b2adf520\") "
Jul 10 19:40:37.033657 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.033638    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772265d4-cc7b-4d1b-bd74-a158b2adf520-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "772265d4-cc7b-4d1b-bd74-a158b2adf520" (UID: "772265d4-cc7b-4d1b-bd74-a158b2adf520"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:40:37.033911 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.033879    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772265d4-cc7b-4d1b-bd74-a158b2adf520-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "772265d4-cc7b-4d1b-bd74-a158b2adf520" (UID: "772265d4-cc7b-4d1b-bd74-a158b2adf520"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:40:37.035451 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.035429    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "772265d4-cc7b-4d1b-bd74-a158b2adf520" (UID: "772265d4-cc7b-4d1b-bd74-a158b2adf520"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:40:37.035556 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.035504    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772265d4-cc7b-4d1b-bd74-a158b2adf520-kube-api-access-79wtf" (OuterVolumeSpecName: "kube-api-access-79wtf") pod "772265d4-cc7b-4d1b-bd74-a158b2adf520" (UID: "772265d4-cc7b-4d1b-bd74-a158b2adf520"). InnerVolumeSpecName "kube-api-access-79wtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:40:37.134520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.134497    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772265d4-cc7b-4d1b-bd74-a158b2adf520-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:37.134520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.134517    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wtf\" (UniqueName: \"kubernetes.io/projected/772265d4-cc7b-4d1b-bd74-a158b2adf520-kube-api-access-79wtf\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:37.134646 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.134526    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772265d4-cc7b-4d1b-bd74-a158b2adf520-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:37.134646 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.134537    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772265d4-cc7b-4d1b-bd74-a158b2adf520-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:40:37.354250 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354213    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"]
Jul 10 19:40:37.354527 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354511    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="storage-initializer"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354530    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="storage-initializer"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354548    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kube-rbac-proxy"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354556    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kube-rbac-proxy"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354574    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354584    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354596    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerName="storage-initializer"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354605    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerName="storage-initializer"
Jul 10 19:40:37.354614 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354615    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerName="storage-initializer"
Jul 10 19:40:37.355053 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354624    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerName="storage-initializer"
Jul 10 19:40:37.355053 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354686    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerName="storage-initializer"
Jul 10 19:40:37.355053 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354699    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kserve-container"
Jul 10 19:40:37.355053 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354709    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e00ea4e-4ab4-4e03-9ea8-5857c948597c" containerName="kube-rbac-proxy"
Jul 10 19:40:37.355053 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.354854    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520" containerName="storage-initializer"
Jul 10 19:40:37.357830 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.357809    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.360188 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.360160    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 10 19:40:37.360188 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.360180    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 10 19:40:37.360375 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.360180    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:40:37.369281 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.369259    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"]
Jul 10 19:40:37.436383 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.436355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3809b85a-cd6a-47c0-9189-407b0e2e197c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.436499 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.436391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqw2\" (UniqueName: \"kubernetes.io/projected/3809b85a-cd6a-47c0-9189-407b0e2e197c-kube-api-access-6zqw2\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.436499 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.436416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3809b85a-cd6a-47c0-9189-407b0e2e197c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.436499 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.436481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.436641 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.436515    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3809b85a-cd6a-47c0-9189-407b0e2e197c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537191 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537428 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537211    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3809b85a-cd6a-47c0-9189-407b0e2e197c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537428 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqw2\" (UniqueName: \"kubernetes.io/projected/3809b85a-cd6a-47c0-9189-407b0e2e197c-kube-api-access-6zqw2\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537593 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537574    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3809b85a-cd6a-47c0-9189-407b0e2e197c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537921 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537902    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.537994 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.537974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.539529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.539508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3809b85a-cd6a-47c0-9189-407b0e2e197c-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.544892 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.544871    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqw2\" (UniqueName: \"kubernetes.io/projected/3809b85a-cd6a-47c0-9189-407b0e2e197c-kube-api-access-6zqw2\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.669425 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.669401    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:37.793197 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.793172    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"]
Jul 10 19:40:37.795572 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:40:37.795545    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3809b85a_cd6a_47c0_9189_407b0e2e197c.slice/crio-7988e2a66ce0c085f33e2a3768d3b7b6ccdf674bc1f25353e5064ebd0a5d83e0 WatchSource:0}: Error finding container 7988e2a66ce0c085f33e2a3768d3b7b6ccdf674bc1f25353e5064ebd0a5d83e0: Status 404 returned error can't find the container with id 7988e2a66ce0c085f33e2a3768d3b7b6ccdf674bc1f25353e5064ebd0a5d83e0
Jul 10 19:40:37.894066 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.894034    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4_772265d4-cc7b-4d1b-bd74-a158b2adf520/storage-initializer/1.log"
Jul 10 19:40:37.894520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.894130    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"
Jul 10 19:40:37.894520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.894147    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4" event={"ID":"772265d4-cc7b-4d1b-bd74-a158b2adf520","Type":"ContainerDied","Data":"680c9e71d9cc761b1345dcd17df23e64891ffe17543032a2d38b2921f5b3b9ba"}
Jul 10 19:40:37.894520 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.894220    2558 scope.go:117] "RemoveContainer" containerID="c59a2cbec80ce617383a8067f07c96796171b55808d8abeb8cb824453d3c7084"
Jul 10 19:40:37.895965 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.895937    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerStarted","Data":"6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c"}
Jul 10 19:40:37.896107 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.895996    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerStarted","Data":"7988e2a66ce0c085f33e2a3768d3b7b6ccdf674bc1f25353e5064ebd0a5d83e0"}
Jul 10 19:40:37.947260 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.947229    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"]
Jul 10 19:40:37.953439 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:37.953414    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-78689458f7-wmxn4"]
Jul 10 19:40:38.901649 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:38.901612    2558 generic.go:356] "Generic (PLEG): container finished" podID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerID="6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c" exitCode=0
Jul 10 19:40:38.902135 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:38.901665    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerDied","Data":"6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c"}
Jul 10 19:40:39.257504 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:39.257472    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772265d4-cc7b-4d1b-bd74-a158b2adf520" path="/var/lib/kubelet/pods/772265d4-cc7b-4d1b-bd74-a158b2adf520/volumes"
Jul 10 19:40:39.907225 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:39.907194    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerStarted","Data":"16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1"}
Jul 10 19:40:39.907621 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:39.907250    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerStarted","Data":"2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd"}
Jul 10 19:40:39.907621 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:39.907364    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:39.927490 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:39.927441    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podStartSLOduration=1.993400593 podStartE2EDuration="2.927427766s" totalImagesPullingTime="0s" totalInitContainerRuntime="934.027173ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:40:39.925858833 +0000 UTC m=+3685.187523607" watchObservedRunningTime="2026-07-10 19:40:39.927427766 +0000 UTC m=+3685.189092539"
Jul 10 19:40:40.911835 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:40.911807    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:40.913078 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:40.913053    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:40:41.914656 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:41.914624    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:40:46.919199 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:46.919167    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:40:46.919694 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:46.919666    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:40:56.919979 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:40:56.919937    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:41:06.919797 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:06.919760    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:41:16.920159 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:16.920121    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:41:26.920541 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:26.920506    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:41:36.919628 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:36.919591    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.63:8080: connect: connection refused"
Jul 10 19:41:46.919883 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:46.919855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:41:47.399327 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:47.399294    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"]
Jul 10 19:41:47.399586 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:47.399560    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container" containerID="cri-o://2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd" gracePeriod=30
Jul 10 19:41:47.399657 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:47.399628    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kube-rbac-proxy" containerID="cri-o://16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1" gracePeriod=30
Jul 10 19:41:48.142690 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.142661    2558 generic.go:356] "Generic (PLEG): container finished" podID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerID="16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1" exitCode=2
Jul 10 19:41:48.143167 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.142757    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerDied","Data":"16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1"}
Jul 10 19:41:48.463765 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.463672    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"]
Jul 10 19:41:48.466281 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.466264    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.470405 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.470381    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 10 19:41:48.470530 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.470465    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 10 19:41:48.487296 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.487275    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"]
Jul 10 19:41:48.549236 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.549211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.549370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.549251    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.549370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.549290    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.549497 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.549370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kube-api-access-jjjx8\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.650440 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.650405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.650591 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.650450    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.650591 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.650489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.650591 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.650512    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kube-api-access-jjjx8\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.650807 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:41:48.650664    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 10 19:41:48.650807 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:41:48.650726    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls podName:cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9 nodeName:}" failed. No retries permitted until 2026-07-10 19:41:49.150710771 +0000 UTC m=+3754.412375527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" (UID: "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Jul 10 19:41:48.650926 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.650860    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.651241 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.651219    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:48.659178 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:48.659154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kube-api-access-jjjx8\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:49.153782 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:49.153754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:49.156150 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:49.156128    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:49.376335 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:49.376300    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:49.506829 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:49.506795    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"]
Jul 10 19:41:49.509974 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:41:49.509944    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf54f7d7_dadf_47c6_8a8a_87c81bc5b2b9.slice/crio-8b591c91a3ed026e3ac04390e38693ebd904b03b91db63df179dfd1c9528b793 WatchSource:0}: Error finding container 8b591c91a3ed026e3ac04390e38693ebd904b03b91db63df179dfd1c9528b793: Status 404 returned error can't find the container with id 8b591c91a3ed026e3ac04390e38693ebd904b03b91db63df179dfd1c9528b793
Jul 10 19:41:50.153888 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:50.153852    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" event={"ID":"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9","Type":"ContainerStarted","Data":"f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab"}
Jul 10 19:41:50.154363 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:50.153911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" event={"ID":"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9","Type":"ContainerStarted","Data":"8b591c91a3ed026e3ac04390e38693ebd904b03b91db63df179dfd1c9528b793"}
Jul 10 19:41:51.642052 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.642025    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:41:51.772211 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772186    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zqw2\" (UniqueName: \"kubernetes.io/projected/3809b85a-cd6a-47c0-9189-407b0e2e197c-kube-api-access-6zqw2\") pod \"3809b85a-cd6a-47c0-9189-407b0e2e197c\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") "
Jul 10 19:41:51.772397 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772222    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3809b85a-cd6a-47c0-9189-407b0e2e197c-proxy-tls\") pod \"3809b85a-cd6a-47c0-9189-407b0e2e197c\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") "
Jul 10 19:41:51.772397 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772241    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"3809b85a-cd6a-47c0-9189-407b0e2e197c\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") "
Jul 10 19:41:51.772397 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772260    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-cabundle-cert\") pod \"3809b85a-cd6a-47c0-9189-407b0e2e197c\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") "
Jul 10 19:41:51.772397 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772299    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3809b85a-cd6a-47c0-9189-407b0e2e197c-kserve-provision-location\") pod \"3809b85a-cd6a-47c0-9189-407b0e2e197c\" (UID: \"3809b85a-cd6a-47c0-9189-407b0e2e197c\") "
Jul 10 19:41:51.772671 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772639    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3809b85a-cd6a-47c0-9189-407b0e2e197c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3809b85a-cd6a-47c0-9189-407b0e2e197c" (UID: "3809b85a-cd6a-47c0-9189-407b0e2e197c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:41:51.772671 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772646    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3809b85a-cd6a-47c0-9189-407b0e2e197c" (UID: "3809b85a-cd6a-47c0-9189-407b0e2e197c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:41:51.772887 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.772702    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "3809b85a-cd6a-47c0-9189-407b0e2e197c" (UID: "3809b85a-cd6a-47c0-9189-407b0e2e197c"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:41:51.774434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.774402    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3809b85a-cd6a-47c0-9189-407b0e2e197c-kube-api-access-6zqw2" (OuterVolumeSpecName: "kube-api-access-6zqw2") pod "3809b85a-cd6a-47c0-9189-407b0e2e197c" (UID: "3809b85a-cd6a-47c0-9189-407b0e2e197c"). InnerVolumeSpecName "kube-api-access-6zqw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:41:51.774495 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.774478    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3809b85a-cd6a-47c0-9189-407b0e2e197c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3809b85a-cd6a-47c0-9189-407b0e2e197c" (UID: "3809b85a-cd6a-47c0-9189-407b0e2e197c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:41:51.872912 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.872884    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3809b85a-cd6a-47c0-9189-407b0e2e197c-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:41:51.872912 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.872908    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqw2\" (UniqueName: \"kubernetes.io/projected/3809b85a-cd6a-47c0-9189-407b0e2e197c-kube-api-access-6zqw2\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:41:51.873069 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.872919    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3809b85a-cd6a-47c0-9189-407b0e2e197c-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:41:51.873069 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.872934    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:41:51.873069 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:51.872944    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3809b85a-cd6a-47c0-9189-407b0e2e197c-cabundle-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:41:52.163259 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.163174    2558 generic.go:356] "Generic (PLEG): container finished" podID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerID="2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd" exitCode=0
Jul 10 19:41:52.163259 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.163230    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"
Jul 10 19:41:52.163439 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.163259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerDied","Data":"2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd"}
Jul 10 19:41:52.163439 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.163328    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b" event={"ID":"3809b85a-cd6a-47c0-9189-407b0e2e197c","Type":"ContainerDied","Data":"7988e2a66ce0c085f33e2a3768d3b7b6ccdf674bc1f25353e5064ebd0a5d83e0"}
Jul 10 19:41:52.163439 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.163369    2558 scope.go:117] "RemoveContainer" containerID="16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1"
Jul 10 19:41:52.171597 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.171584    2558 scope.go:117] "RemoveContainer" containerID="2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd"
Jul 10 19:41:52.178951 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.178935    2558 scope.go:117] "RemoveContainer" containerID="6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c"
Jul 10 19:41:52.184071 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.184051    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"]
Jul 10 19:41:52.186544 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.186530    2558 scope.go:117] "RemoveContainer" containerID="16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1"
Jul 10 19:41:52.186865 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:41:52.186847    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1\": container with ID starting with 16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1 not found: ID does not exist" containerID="16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1"
Jul 10 19:41:52.186951 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.186878    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1"} err="failed to get container status \"16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1\": rpc error: code = NotFound desc = could not find container \"16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1\": container with ID starting with 16027994026a7ce5fb6fabadf47d1f8baf4932ca153a9a10ec7a59dba135f4c1 not found: ID does not exist"
Jul 10 19:41:52.186951 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.186902    2558 scope.go:117] "RemoveContainer" containerID="2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd"
Jul 10 19:41:52.186951 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.186921    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-98ffcd57f-npq4b"]
Jul 10 19:41:52.187167 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:41:52.187149    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd\": container with ID starting with 2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd not found: ID does not exist" containerID="2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd"
Jul 10 19:41:52.187210 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.187173    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd"} err="failed to get container status \"2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd\": rpc error: code = NotFound desc = could not find container \"2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd\": container with ID starting with 2096bb7279d8180ed4d074d86b5fe9eb5f7f5c912bc77d55bb9767a563846dfd not found: ID does not exist"
Jul 10 19:41:52.187210 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.187189    2558 scope.go:117] "RemoveContainer" containerID="6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c"
Jul 10 19:41:52.187400 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:41:52.187384    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c\": container with ID starting with 6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c not found: ID does not exist" containerID="6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c"
Jul 10 19:41:52.187444 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:52.187405    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c"} err="failed to get container status \"6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c\": rpc error: code = NotFound desc = could not find container \"6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c\": container with ID starting with 6e9e12cea237f177da53e1206ae2349aec2bdf45d75ce0e63863054a76192b5c not found: ID does not exist"
Jul 10 19:41:53.256699 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:53.256668    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" path="/var/lib/kubelet/pods/3809b85a-cd6a-47c0-9189-407b0e2e197c/volumes"
Jul 10 19:41:54.172592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:54.172522    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6_cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9/storage-initializer/0.log"
Jul 10 19:41:54.172592 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:54.172558    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerID="f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab" exitCode=1
Jul 10 19:41:54.172805 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:54.172618    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" event={"ID":"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9","Type":"ContainerDied","Data":"f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab"}
Jul 10 19:41:55.177982 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:55.177954    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6_cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9/storage-initializer/0.log"
Jul 10 19:41:55.178329 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:55.178025    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" event={"ID":"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9","Type":"ContainerStarted","Data":"88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9"}
Jul 10 19:41:58.459130 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:58.459096    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"]
Jul 10 19:41:58.459514 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:58.459314    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerName="storage-initializer" containerID="cri-o://88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9" gracePeriod=30
Jul 10 19:41:59.544499 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544463    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"]
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544761    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kube-rbac-proxy"
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544772    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kube-rbac-proxy"
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544783    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="storage-initializer"
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544789    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="storage-initializer"
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544800    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container"
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544807    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container"
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544854    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kube-rbac-proxy"
Jul 10 19:41:59.544904 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.544861    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3809b85a-cd6a-47c0-9189-407b0e2e197c" containerName="kserve-container"
Jul 10 19:41:59.546927 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.546908    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.549309 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.549268    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 10 19:41:59.549309 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.549277    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 10 19:41:59.549512 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.549376    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 10 19:41:59.560043 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.560012    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"]
Jul 10 19:41:59.729272 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.729242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.729400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.729283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.729400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.729311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee6d1b5-e30a-482a-b469-02b65ac9d849-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.729400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.729363    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.729400 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.729382    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kztxn\" (UniqueName: \"kubernetes.io/projected/dee6d1b5-e30a-482a-b469-02b65ac9d849-kube-api-access-kztxn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.802846 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.802793    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6_cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9/storage-initializer/1.log"
Jul 10 19:41:59.803142 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.803128    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6_cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9/storage-initializer/0.log"
Jul 10 19:41:59.804778 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.804759    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:41:59.830234 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830209    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.830366 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kztxn\" (UniqueName: \"kubernetes.io/projected/dee6d1b5-e30a-482a-b469-02b65ac9d849-kube-api-access-kztxn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.830366 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.830366 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.830528 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee6d1b5-e30a-482a-b469-02b65ac9d849-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.830528 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:41:59.830462    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 10 19:41:59.830638 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:41:59.830529    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls podName:dee6d1b5-e30a-482a-b469-02b65ac9d849 nodeName:}" failed. No retries permitted until 2026-07-10 19:42:00.330507812 +0000 UTC m=+3765.592172565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" (UID: "dee6d1b5-e30a-482a-b469-02b65ac9d849") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Jul 10 19:41:59.830798 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee6d1b5-e30a-482a-b469-02b65ac9d849-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.830980 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.831016 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.830975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.840112 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.840093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kztxn\" (UniqueName: \"kubernetes.io/projected/dee6d1b5-e30a-482a-b469-02b65ac9d849-kube-api-access-kztxn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:41:59.930756 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.930720    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kube-api-access-jjjx8\") pod \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") "
Jul 10 19:41:59.930879 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.930818    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls\") pod \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") "
Jul 10 19:41:59.930939 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.930874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") "
Jul 10 19:41:59.930939 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.930908    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kserve-provision-location\") pod \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\" (UID: \"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9\") "
Jul 10 19:41:59.936635 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.931553    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" (UID: "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:41:59.936635 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.931558    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" (UID: "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:41:59.936635 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.933461    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kube-api-access-jjjx8" (OuterVolumeSpecName: "kube-api-access-jjjx8") pod "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" (UID: "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9"). InnerVolumeSpecName "kube-api-access-jjjx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:41:59.936635 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:41:59.933528    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" (UID: "cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:42:00.032654 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.032623    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjx8\" (UniqueName: \"kubernetes.io/projected/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kube-api-access-jjjx8\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:42:00.032654 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.032649    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:42:00.032654 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.032659    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:42:00.032917 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.032669    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:42:00.198344 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.198263    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6_cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9/storage-initializer/1.log"
Jul 10 19:42:00.198666 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.198647    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6_cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9/storage-initializer/0.log"
Jul 10 19:42:00.198767 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.198685    2558 generic.go:356] "Generic (PLEG): container finished" podID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerID="88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9" exitCode=1
Jul 10 19:42:00.198767 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.198717    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" event={"ID":"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9","Type":"ContainerDied","Data":"88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9"}
Jul 10 19:42:00.198893 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.198783    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"
Jul 10 19:42:00.198893 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.198826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6" event={"ID":"cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9","Type":"ContainerDied","Data":"8b591c91a3ed026e3ac04390e38693ebd904b03b91db63df179dfd1c9528b793"}
Jul 10 19:42:00.198893 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.198855    2558 scope.go:117] "RemoveContainer" containerID="88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9"
Jul 10 19:42:00.207030 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.206924    2558 scope.go:117] "RemoveContainer" containerID="f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab"
Jul 10 19:42:00.214026 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.214010    2558 scope.go:117] "RemoveContainer" containerID="88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9"
Jul 10 19:42:00.214284 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:42:00.214267    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9\": container with ID starting with 88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9 not found: ID does not exist" containerID="88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9"
Jul 10 19:42:00.214324 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.214294    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9"} err="failed to get container status \"88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9\": rpc error: code = NotFound desc = could not find container \"88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9\": container with ID starting with 88caee9d73a4b9f56bf69e76c7847b6f352b70fe243a961d9cdf28be72e4b0f9 not found: ID does not exist"
Jul 10 19:42:00.214324 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.214310    2558 scope.go:117] "RemoveContainer" containerID="f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab"
Jul 10 19:42:00.214539 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:42:00.214523    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab\": container with ID starting with f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab not found: ID does not exist" containerID="f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab"
Jul 10 19:42:00.214579 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.214546    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab"} err="failed to get container status \"f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab\": rpc error: code = NotFound desc = could not find container \"f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab\": container with ID starting with f387283a2813b3b5de6c8c027eb0a928b1f3f048dc208f9af96496bdec8f85ab not found: ID does not exist"
Jul 10 19:42:00.233479 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.233454    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"]
Jul 10 19:42:00.237367 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.237345    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-75985686d5-vt2g6"]
Jul 10 19:42:00.334644 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.334611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:42:00.337038 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.337009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:42:00.457142 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.457056    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:42:00.584183 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:00.584155    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"]
Jul 10 19:42:00.587385 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:42:00.587359    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee6d1b5_e30a_482a_b469_02b65ac9d849.slice/crio-60087820ac17443e23fb4a893c161297535407d79c8fff06a7f721b4b4fa6498 WatchSource:0}: Error finding container 60087820ac17443e23fb4a893c161297535407d79c8fff06a7f721b4b4fa6498: Status 404 returned error can't find the container with id 60087820ac17443e23fb4a893c161297535407d79c8fff06a7f721b4b4fa6498
Jul 10 19:42:01.205418 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:01.205383    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerStarted","Data":"ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734"}
Jul 10 19:42:01.205596 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:01.205455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerStarted","Data":"60087820ac17443e23fb4a893c161297535407d79c8fff06a7f721b4b4fa6498"}
Jul 10 19:42:01.256529 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:01.256486    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" path="/var/lib/kubelet/pods/cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9/volumes"
Jul 10 19:42:02.210875 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:02.210838    2558 generic.go:356] "Generic (PLEG): container finished" podID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerID="ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734" exitCode=0
Jul 10 19:42:02.211264 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:02.210899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerDied","Data":"ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734"}
Jul 10 19:42:03.216229 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:03.216195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerStarted","Data":"afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf"}
Jul 10 19:42:03.216722 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:03.216265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerStarted","Data":"6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b"}
Jul 10 19:42:03.216722 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:03.216298    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:42:03.233766 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:03.233680    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podStartSLOduration=3.315854356 podStartE2EDuration="4.233660542s" totalImagesPullingTime="0s" totalInitContainerRuntime="917.806186ms" isStatefulPod=false podCreationTimestamp="2026-07-10 19:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:42:03.232830046 +0000 UTC m=+3768.494494887" watchObservedRunningTime="2026-07-10 19:42:03.233660542 +0000 UTC m=+3768.495325316"
Jul 10 19:42:04.219599 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:04.219566    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:42:04.220866 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:04.220840    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:42:05.223070 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:05.223033    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:42:10.227684 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:10.227656    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:42:10.228275 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:10.228247    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:42:20.228961 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:20.228870    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:42:30.228587 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:30.228550    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:42:40.228850 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:40.228807    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:42:50.228883 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:42:50.228843    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:43:00.229076 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:00.229039    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:43:10.228805 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:10.228776    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:43:19.604429 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:19.604397    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"]
Jul 10 19:43:19.604852 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:19.604625    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" containerID="cri-o://6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b" gracePeriod=30
Jul 10 19:43:19.604852 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:19.604674    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kube-rbac-proxy" containerID="cri-o://afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf" gracePeriod=30
Jul 10 19:43:20.223392 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.223349    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Jul 10 19:43:20.228681 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.228657    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Jul 10 19:43:20.489122 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.489029    2558 generic.go:356] "Generic (PLEG): container finished" podID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerID="afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf" exitCode=2
Jul 10 19:43:20.489270 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.489108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerDied","Data":"afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf"}
Jul 10 19:43:20.637762 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.637715    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"]
Jul 10 19:43:20.638125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.638005    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerName="storage-initializer"
Jul 10 19:43:20.638125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.638016    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerName="storage-initializer"
Jul 10 19:43:20.638125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.638069    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerName="storage-initializer"
Jul 10 19:43:20.638125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.638075    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerName="storage-initializer"
Jul 10 19:43:20.638125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.638123    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerName="storage-initializer"
Jul 10 19:43:20.638125 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.638130    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf54f7d7-dadf-47c6-8a8a-87c81bc5b2b9" containerName="storage-initializer"
Jul 10 19:43:20.640885 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.640861    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.643116 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.643087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 10 19:43:20.643260 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.643238    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 10 19:43:20.653056 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.653031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"]
Jul 10 19:43:20.805578 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.805551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrmw\" (UniqueName: \"kubernetes.io/projected/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kube-api-access-msrmw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.805735 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.805586    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.805735 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.805623    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.805735 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.805650    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.907018 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.906984    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.907155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.907029    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.907155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.907051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.907155 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.907098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrmw\" (UniqueName: \"kubernetes.io/projected/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kube-api-access-msrmw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.907306 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:20.907164    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:20.907306 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:20.907234    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls podName:f2af2ccf-f251-416b-a0a6-4667a5db2fb7 nodeName:}" failed. No retries permitted until 2026-07-10 19:43:21.407212206 +0000 UTC m=+3846.668876968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:20.907622 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.907601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.907755 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.907723    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:20.915349 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:20.915328    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrmw\" (UniqueName: \"kubernetes.io/projected/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kube-api-access-msrmw\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:21.410776 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:21.410727    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:21.413113 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:21.413086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:21.551159 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:21.551129    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:21.675797 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:21.675711    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"]
Jul 10 19:43:21.678924 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:43:21.678896    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2af2ccf_f251_416b_a0a6_4667a5db2fb7.slice/crio-9a5da54cbd7eabfdf8f0cac8d353f471f52296c179c0ecd4db1e346132921e0f WatchSource:0}: Error finding container 9a5da54cbd7eabfdf8f0cac8d353f471f52296c179c0ecd4db1e346132921e0f: Status 404 returned error can't find the container with id 9a5da54cbd7eabfdf8f0cac8d353f471f52296c179c0ecd4db1e346132921e0f
Jul 10 19:43:22.497223 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:22.497188    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" event={"ID":"f2af2ccf-f251-416b-a0a6-4667a5db2fb7","Type":"ContainerStarted","Data":"9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed"}
Jul 10 19:43:22.497403 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:22.497248    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" event={"ID":"f2af2ccf-f251-416b-a0a6-4667a5db2fb7","Type":"ContainerStarted","Data":"9a5da54cbd7eabfdf8f0cac8d353f471f52296c179c0ecd4db1e346132921e0f"}
Jul 10 19:43:23.743708 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.743685    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:43:23.827983 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.827922    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls\") pod \"dee6d1b5-e30a-482a-b469-02b65ac9d849\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") "
Jul 10 19:43:23.827983 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.827957    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"dee6d1b5-e30a-482a-b469-02b65ac9d849\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") "
Jul 10 19:43:23.828176 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.827995    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee6d1b5-e30a-482a-b469-02b65ac9d849-kserve-provision-location\") pod \"dee6d1b5-e30a-482a-b469-02b65ac9d849\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") "
Jul 10 19:43:23.828176 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.828089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-cabundle-cert\") pod \"dee6d1b5-e30a-482a-b469-02b65ac9d849\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") "
Jul 10 19:43:23.828176 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.828147    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kztxn\" (UniqueName: \"kubernetes.io/projected/dee6d1b5-e30a-482a-b469-02b65ac9d849-kube-api-access-kztxn\") pod \"dee6d1b5-e30a-482a-b469-02b65ac9d849\" (UID: \"dee6d1b5-e30a-482a-b469-02b65ac9d849\") "
Jul 10 19:43:23.828370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.828346    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee6d1b5-e30a-482a-b469-02b65ac9d849-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dee6d1b5-e30a-482a-b469-02b65ac9d849" (UID: "dee6d1b5-e30a-482a-b469-02b65ac9d849"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:43:23.828427 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.828401    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dee6d1b5-e30a-482a-b469-02b65ac9d849" (UID: "dee6d1b5-e30a-482a-b469-02b65ac9d849"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:43:23.828427 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.828398    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "dee6d1b5-e30a-482a-b469-02b65ac9d849" (UID: "dee6d1b5-e30a-482a-b469-02b65ac9d849"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:43:23.830028 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.830005    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dee6d1b5-e30a-482a-b469-02b65ac9d849" (UID: "dee6d1b5-e30a-482a-b469-02b65ac9d849"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:43:23.830159 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.830141    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee6d1b5-e30a-482a-b469-02b65ac9d849-kube-api-access-kztxn" (OuterVolumeSpecName: "kube-api-access-kztxn") pod "dee6d1b5-e30a-482a-b469-02b65ac9d849" (UID: "dee6d1b5-e30a-482a-b469-02b65ac9d849"). InnerVolumeSpecName "kube-api-access-kztxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:43:23.929013 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.928990    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee6d1b5-e30a-482a-b469-02b65ac9d849-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:23.929013 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.929011    2558 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-cabundle-cert\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:23.929157 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.929022    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kztxn\" (UniqueName: \"kubernetes.io/projected/dee6d1b5-e30a-482a-b469-02b65ac9d849-kube-api-access-kztxn\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:23.929157 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.929030    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dee6d1b5-e30a-482a-b469-02b65ac9d849-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:23.929157 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:23.929039    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dee6d1b5-e30a-482a-b469-02b65ac9d849-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:24.505496 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.505463    2558 generic.go:356] "Generic (PLEG): container finished" podID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerID="6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b" exitCode=0
Jul 10 19:43:24.505671 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.505536    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"
Jul 10 19:43:24.505671 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.505539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerDied","Data":"6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b"}
Jul 10 19:43:24.505775 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.505686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr" event={"ID":"dee6d1b5-e30a-482a-b469-02b65ac9d849","Type":"ContainerDied","Data":"60087820ac17443e23fb4a893c161297535407d79c8fff06a7f721b4b4fa6498"}
Jul 10 19:43:24.505775 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.505713    2558 scope.go:117] "RemoveContainer" containerID="afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf"
Jul 10 19:43:24.514020 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.514002    2558 scope.go:117] "RemoveContainer" containerID="6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b"
Jul 10 19:43:24.520814 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.520793    2558 scope.go:117] "RemoveContainer" containerID="ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734"
Jul 10 19:43:24.525590 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.525569    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"]
Jul 10 19:43:24.528209 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.528192    2558 scope.go:117] "RemoveContainer" containerID="afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf"
Jul 10 19:43:24.528464 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:24.528443    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf\": container with ID starting with afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf not found: ID does not exist" containerID="afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf"
Jul 10 19:43:24.528526 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.528471    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf"} err="failed to get container status \"afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf\": rpc error: code = NotFound desc = could not find container \"afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf\": container with ID starting with afa0172b816e33f785d0758328d9daa9ac3003caad7535ff6295e37d26e374cf not found: ID does not exist"
Jul 10 19:43:24.528526 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.528490    2558 scope.go:117] "RemoveContainer" containerID="6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b"
Jul 10 19:43:24.528773 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:24.528728    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b\": container with ID starting with 6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b not found: ID does not exist" containerID="6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b"
Jul 10 19:43:24.528843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.528782    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b"} err="failed to get container status \"6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b\": rpc error: code = NotFound desc = could not find container \"6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b\": container with ID starting with 6fd70649afa10914919e22b7153fc93044e80929c7c073d746cdf26b3900120b not found: ID does not exist"
Jul 10 19:43:24.528843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.528804    2558 scope.go:117] "RemoveContainer" containerID="ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734"
Jul 10 19:43:24.529063 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:24.529047    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734\": container with ID starting with ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734 not found: ID does not exist" containerID="ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734"
Jul 10 19:43:24.529101 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.529067    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734"} err="failed to get container status \"ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734\": rpc error: code = NotFound desc = could not find container \"ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734\": container with ID starting with ed1c47f8784438d607c39ba8a46f3852eefd20e67e5f91ba1562863b8c951734 not found: ID does not exist"
Jul 10 19:43:24.529310 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:24.529295    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-d9fd95c58-jxscr"]
Jul 10 19:43:25.257044 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:25.257011    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" path="/var/lib/kubelet/pods/dee6d1b5-e30a-482a-b469-02b65ac9d849/volumes"
Jul 10 19:43:29.527679 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:29.527650    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5_f2af2ccf-f251-416b-a0a6-4667a5db2fb7/storage-initializer/0.log"
Jul 10 19:43:29.528099 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:29.527687    2558 generic.go:356] "Generic (PLEG): container finished" podID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerID="9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed" exitCode=1
Jul 10 19:43:29.528099 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:29.527730    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" event={"ID":"f2af2ccf-f251-416b-a0a6-4667a5db2fb7","Type":"ContainerDied","Data":"9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed"}
Jul 10 19:43:30.534360 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:30.534337    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5_f2af2ccf-f251-416b-a0a6-4667a5db2fb7/storage-initializer/0.log"
Jul 10 19:43:30.534771 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:30.534426    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" event={"ID":"f2af2ccf-f251-416b-a0a6-4667a5db2fb7","Type":"ContainerStarted","Data":"6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec"}
Jul 10 19:43:30.646030 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:30.645997    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"]
Jul 10 19:43:30.679616 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:30.679585    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:30.679760 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:30.679663    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls podName:f2af2ccf-f251-416b-a0a6-4667a5db2fb7 nodeName:}" failed. No retries permitted until 2026-07-10 19:43:31.179641353 +0000 UTC m=+3856.441306115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:31.183180 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:31.183149    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:31.183420 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:31.183220    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls podName:f2af2ccf-f251-416b-a0a6-4667a5db2fb7 nodeName:}" failed. No retries permitted until 2026-07-10 19:43:32.183203148 +0000 UTC m=+3857.444867915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:31.537576 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:31.537543    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerName="storage-initializer" containerID="cri-o://6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec" gracePeriod=30
Jul 10 19:43:32.191579 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:32.191541    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:32.191784 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:32.191622    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls podName:f2af2ccf-f251-416b-a0a6-4667a5db2fb7 nodeName:}" failed. No retries permitted until 2026-07-10 19:43:34.191607757 +0000 UTC m=+3859.453272507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:34.178463 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.178441    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5_f2af2ccf-f251-416b-a0a6-4667a5db2fb7/storage-initializer/1.log"
Jul 10 19:43:34.178823 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.178810    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5_f2af2ccf-f251-416b-a0a6-4667a5db2fb7/storage-initializer/0.log"
Jul 10 19:43:34.180947 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.180931    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:34.206632 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:34.206609    2558 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:34.206771 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:34.206663    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls podName:f2af2ccf-f251-416b-a0a6-4667a5db2fb7 nodeName:}" failed. No retries permitted until 2026-07-10 19:43:38.206651311 +0000 UTC m=+3863.468316062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 10 19:43:34.307507 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.307429    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls\") pod \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") "
Jul 10 19:43:34.307507 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.307471    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msrmw\" (UniqueName: \"kubernetes.io/projected/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kube-api-access-msrmw\") pod \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") "
Jul 10 19:43:34.307507 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.307496    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") "
Jul 10 19:43:34.307787 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.307557    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kserve-provision-location\") pod \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\" (UID: \"f2af2ccf-f251-416b-a0a6-4667a5db2fb7\") "
Jul 10 19:43:34.307883 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.307858    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2af2ccf-f251-416b-a0a6-4667a5db2fb7" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 19:43:34.307930 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.307899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "f2af2ccf-f251-416b-a0a6-4667a5db2fb7" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 19:43:34.309513 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.309490    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2af2ccf-f251-416b-a0a6-4667a5db2fb7" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 19:43:34.309617 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.309562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kube-api-access-msrmw" (OuterVolumeSpecName: "kube-api-access-msrmw") pod "f2af2ccf-f251-416b-a0a6-4667a5db2fb7" (UID: "f2af2ccf-f251-416b-a0a6-4667a5db2fb7"). InnerVolumeSpecName "kube-api-access-msrmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 19:43:34.408468 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.408445    2558 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kserve-provision-location\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:34.408468 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.408468    2558 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-proxy-tls\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:34.408628 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.408478    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrmw\" (UniqueName: \"kubernetes.io/projected/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-kube-api-access-msrmw\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:34.408628 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.408493    2558 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2af2ccf-f251-416b-a0a6-4667a5db2fb7-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-163.ec2.internal\" DevicePath \"\""
Jul 10 19:43:34.548305 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.548281    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5_f2af2ccf-f251-416b-a0a6-4667a5db2fb7/storage-initializer/1.log"
Jul 10 19:43:34.548653 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.548638    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5_f2af2ccf-f251-416b-a0a6-4667a5db2fb7/storage-initializer/0.log"
Jul 10 19:43:34.548711 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.548674    2558 generic.go:356] "Generic (PLEG): container finished" podID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerID="6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec" exitCode=1
Jul 10 19:43:34.548784 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.548710    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" event={"ID":"f2af2ccf-f251-416b-a0a6-4667a5db2fb7","Type":"ContainerDied","Data":"6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec"}
Jul 10 19:43:34.548784 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.548737    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"
Jul 10 19:43:34.548910 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.548807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5" event={"ID":"f2af2ccf-f251-416b-a0a6-4667a5db2fb7","Type":"ContainerDied","Data":"9a5da54cbd7eabfdf8f0cac8d353f471f52296c179c0ecd4db1e346132921e0f"}
Jul 10 19:43:34.548910 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.548851    2558 scope.go:117] "RemoveContainer" containerID="6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec"
Jul 10 19:43:34.557434 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.557411    2558 scope.go:117] "RemoveContainer" containerID="9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed"
Jul 10 19:43:34.564179 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.564162    2558 scope.go:117] "RemoveContainer" containerID="6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec"
Jul 10 19:43:34.564423 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:34.564406    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec\": container with ID starting with 6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec not found: ID does not exist" containerID="6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec"
Jul 10 19:43:34.564466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.564432    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec"} err="failed to get container status \"6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec\": rpc error: code = NotFound desc = could not find container \"6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec\": container with ID starting with 6aa8d378eaef42dce7f04fd219ce07bff8bdc4f90e300930f84a9cbdf37d52ec not found: ID does not exist"
Jul 10 19:43:34.564466 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.564449    2558 scope.go:117] "RemoveContainer" containerID="9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed"
Jul 10 19:43:34.564665 ip-10-0-140-163 kubenswrapper[2558]: E0710 19:43:34.564650    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed\": container with ID starting with 9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed not found: ID does not exist" containerID="9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed"
Jul 10 19:43:34.564701 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.564671    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed"} err="failed to get container status \"9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed\": rpc error: code = NotFound desc = could not find container \"9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed\": container with ID starting with 9fdb3e045443dda5c621b36ec417e1e3e5d0e69f0040b88315a1bdc4425b84ed not found: ID does not exist"
Jul 10 19:43:34.586852 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.586828    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"]
Jul 10 19:43:34.590598 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:34.590580    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6f7499d8bd-q99c5"]
Jul 10 19:43:35.256584 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:43:35.256552    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" path="/var/lib/kubelet/pods/f2af2ccf-f251-416b-a0a6-4667a5db2fb7/volumes"
Jul 10 19:44:04.274546 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274510    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vwzv/must-gather-t9bcj"]
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274927    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274945    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274956    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kube-rbac-proxy"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274964    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kube-rbac-proxy"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274976    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274985    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.274998    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.275006    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.275014    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.275023    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.275087    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kserve-container"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.275100    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.275110    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2af2ccf-f251-416b-a0a6-4667a5db2fb7" containerName="storage-initializer"
Jul 10 19:44:04.275160 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.275119    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee6d1b5-e30a-482a-b469-02b65ac9d849" containerName="kube-rbac-proxy"
Jul 10 19:44:04.277956 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.277935    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.280192 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.280168    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vwzv\"/\"openshift-service-ca.crt\""
Jul 10 19:44:04.280667 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.280647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vwzv\"/\"kube-root-ca.crt\""
Jul 10 19:44:04.281102 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.281087    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vwzv\"/\"default-dockercfg-qr9qg\""
Jul 10 19:44:04.287541 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.287524    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwzv/must-gather-t9bcj"]
Jul 10 19:44:04.311055 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.311030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkdb\" (UniqueName: \"kubernetes.io/projected/8b44690a-baf6-4e1a-a8ff-5f4ffb63600f-kube-api-access-dvkdb\") pod \"must-gather-t9bcj\" (UID: \"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f\") " pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.311163 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.311062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b44690a-baf6-4e1a-a8ff-5f4ffb63600f-must-gather-output\") pod \"must-gather-t9bcj\" (UID: \"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f\") " pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.411895 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.411873    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b44690a-baf6-4e1a-a8ff-5f4ffb63600f-must-gather-output\") pod \"must-gather-t9bcj\" (UID: \"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f\") " pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.411991 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.411928    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkdb\" (UniqueName: \"kubernetes.io/projected/8b44690a-baf6-4e1a-a8ff-5f4ffb63600f-kube-api-access-dvkdb\") pod \"must-gather-t9bcj\" (UID: \"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f\") " pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.412177 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.412160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b44690a-baf6-4e1a-a8ff-5f4ffb63600f-must-gather-output\") pod \"must-gather-t9bcj\" (UID: \"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f\") " pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.419882 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.419855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkdb\" (UniqueName: \"kubernetes.io/projected/8b44690a-baf6-4e1a-a8ff-5f4ffb63600f-kube-api-access-dvkdb\") pod \"must-gather-t9bcj\" (UID: \"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f\") " pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.587910 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.587815    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwzv/must-gather-t9bcj"
Jul 10 19:44:04.710471 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.710440    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwzv/must-gather-t9bcj"]
Jul 10 19:44:04.713149 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:44:04.713122    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b44690a_baf6_4e1a_a8ff_5f4ffb63600f.slice/crio-b0e6a1df10f536ed5debf856e321fb849f1c7049dc84a8230c752ba023a35d25 WatchSource:0}: Error finding container b0e6a1df10f536ed5debf856e321fb849f1c7049dc84a8230c752ba023a35d25: Status 404 returned error can't find the container with id b0e6a1df10f536ed5debf856e321fb849f1c7049dc84a8230c752ba023a35d25
Jul 10 19:44:04.714898 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:04.714879    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 19:44:05.659637 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:05.659580    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwzv/must-gather-t9bcj" event={"ID":"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f","Type":"ContainerStarted","Data":"b0e6a1df10f536ed5debf856e321fb849f1c7049dc84a8230c752ba023a35d25"}
Jul 10 19:44:06.665305 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:06.665267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwzv/must-gather-t9bcj" event={"ID":"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f","Type":"ContainerStarted","Data":"bb0ab1c61b16f628712ccdac7348659c63aeb7698e420c40e57355679149ffc9"}
Jul 10 19:44:06.665820 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:06.665319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwzv/must-gather-t9bcj" event={"ID":"8b44690a-baf6-4e1a-a8ff-5f4ffb63600f","Type":"ContainerStarted","Data":"ffc7f93e4b94e0ef714c719c3bbe6e8993a8af61cf6344a9bffa877c31018390"}
Jul 10 19:44:06.681595 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:06.681311    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vwzv/must-gather-t9bcj" podStartSLOduration=1.771461441 podStartE2EDuration="2.681289744s" totalImagesPullingTime="909.828303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:44:06.679867937 +0000 UTC m=+3891.941532710" watchObservedRunningTime="2026-07-10 19:44:06.681289744 +0000 UTC m=+3891.942954519"
Jul 10 19:44:07.182290 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:07.182237    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bcvk5_2d494c4a-3c15-495c-a993-9bbf592b6d50/global-pull-secret-syncer/0.log"
Jul 10 19:44:07.384985 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:07.384953    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sqq97_fae3b187-d48f-43e9-a6a3-8bd2dae78715/konnectivity-agent/0.log"
Jul 10 19:44:07.460486 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:07.460412    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-163.ec2.internal_4084b00d8aece702c3f7f6e211f96825/haproxy/0.log"
Jul 10 19:44:12.124287 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:12.124255    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8dqw_2d75b631-1c18-44f7-8893-0f703c28900d/node-exporter/0.log"
Jul 10 19:44:12.147307 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:12.147278    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8dqw_2d75b631-1c18-44f7-8893-0f703c28900d/kube-rbac-proxy/0.log"
Jul 10 19:44:12.172465 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:12.172436    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g8dqw_2d75b631-1c18-44f7-8893-0f703c28900d/init-textfile/0.log"
Jul 10 19:44:15.440944 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.440895    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"]
Jul 10 19:44:15.445110 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.445080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.459257 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.459224    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"]
Jul 10 19:44:15.490324 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.490286    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:44:15.520179 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.520145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-proc\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.520473 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.520453    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8w2\" (UniqueName: \"kubernetes.io/projected/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-kube-api-access-4t8w2\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.520617 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.520604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-sys\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.520721 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.520709    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-podres\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.520893 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.520875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-lib-modules\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622246 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-proc\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622263    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8w2\" (UniqueName: \"kubernetes.io/projected/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-kube-api-access-4t8w2\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-sys\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622335    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-podres\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-proc\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622432 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622409    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-sys\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-lib-modules\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622532    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-podres\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.622698 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.622586    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-lib-modules\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.630035 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.630003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8w2\" (UniqueName: \"kubernetes.io/projected/e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3-kube-api-access-4t8w2\") pod \"perf-node-gather-daemonset-ntbvl\" (UID: \"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3\") " pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.758385 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.758353    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:15.920612 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:15.920238    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"]
Jul 10 19:44:15.924977 ip-10-0-140-163 kubenswrapper[2558]: W0710 19:44:15.924949    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode3b34ccd_e40a_4b18_8bb9_45ed2b6796e3.slice/crio-1167bab3cfa84329d7be490d89d94cf072f0f2d32e2a801fbdb336bd1d4bf67e WatchSource:0}: Error finding container 1167bab3cfa84329d7be490d89d94cf072f0f2d32e2a801fbdb336bd1d4bf67e: Status 404 returned error can't find the container with id 1167bab3cfa84329d7be490d89d94cf072f0f2d32e2a801fbdb336bd1d4bf67e
Jul 10 19:44:16.663495 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:16.663466    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hhxbq_ef7e1cdc-4e50-4260-9838-df169c0b9ade/dns/0.log"
Jul 10 19:44:16.685440 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:16.685413    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hhxbq_ef7e1cdc-4e50-4260-9838-df169c0b9ade/kube-rbac-proxy/0.log"
Jul 10 19:44:16.715875 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:16.715844    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl" event={"ID":"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3","Type":"ContainerStarted","Data":"cbc8767a92ddf366459c60c5d5327750df3c654967fd852c6a1b0c2c949228ad"}
Jul 10 19:44:16.716064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:16.715899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl" event={"ID":"e3b34ccd-e40a-4b18-8bb9-45ed2b6796e3","Type":"ContainerStarted","Data":"1167bab3cfa84329d7be490d89d94cf072f0f2d32e2a801fbdb336bd1d4bf67e"}
Jul 10 19:44:16.716064 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:16.716005    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:16.732806 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:16.732761    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl" podStartSLOduration=1.732724572 podStartE2EDuration="1.732724572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:44:16.731908665 +0000 UTC m=+3901.993573439" watchObservedRunningTime="2026-07-10 19:44:16.732724572 +0000 UTC m=+3901.994389345"
Jul 10 19:44:16.733575 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:16.733550    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5kd9_88cc0f38-18da-4e79-b96f-5b6435cdf0d7/dns-node-resolver/0.log"
Jul 10 19:44:17.243815 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:17.243789    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jqvjm_fcc1787d-a277-48fe-b6ea-8dde75b794ef/node-ca/0.log"
Jul 10 19:44:18.483370 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:18.483343    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xr5nv_9f070177-34ce-4586-94ed-96206e46da76/serve-healthcheck-canary/0.log"
Jul 10 19:44:18.729126 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:18.729101    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:44:18.987362 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:18.987331    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rvxcl_5f1a2e54-5f36-425c-a557-8b78d8101f1e/kube-rbac-proxy/0.log"
Jul 10 19:44:19.008598 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:19.008573    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rvxcl_5f1a2e54-5f36-425c-a557-8b78d8101f1e/exporter/0.log"
Jul 10 19:44:19.029695 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:19.029672    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rvxcl_5f1a2e54-5f36-425c-a557-8b78d8101f1e/extractor/0.log"
Jul 10 19:44:21.202408 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:21.202371    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-659578d74d-nt8hp_c2329682-ee12-4c91-89f9-b5bf5fee66fa/manager/0.log"
Jul 10 19:44:21.679843 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:21.679791    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-b5kp2_54270212-d6b3-4e62-b52a-74985d4dcd57/manager/0.log"
Jul 10 19:44:21.700542 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:21.700512    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-jp47q_80e5d68b-f539-44c6-95a9-cbc49d705fc3/s3-init/0.log"
Jul 10 19:44:21.777898 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:21.777867    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-vhtrw_d790a2c2-0ccf-4e95-826a-6f3adc27de39/seaweedfs/0.log"
Jul 10 19:44:21.826681 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:21.826654    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-s2b89_896aac54-26bf-485b-82fb-68a2797cdf7f/seaweedfs-tls-serving/0.log"
Jul 10 19:44:22.728754 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:22.728709    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vwzv/perf-node-gather-daemonset-ntbvl"
Jul 10 19:44:27.186097 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.186023    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2dpm8_2a5f5bf2-ff2e-49c2-b688-b1b298de105d/kube-multus-additional-cni-plugins/0.log"
Jul 10 19:44:27.209040 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.208985    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2dpm8_2a5f5bf2-ff2e-49c2-b688-b1b298de105d/egress-router-binary-copy/0.log"
Jul 10 19:44:27.231113 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.231081    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2dpm8_2a5f5bf2-ff2e-49c2-b688-b1b298de105d/cni-plugins/0.log"
Jul 10 19:44:27.259738 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.259716    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2dpm8_2a5f5bf2-ff2e-49c2-b688-b1b298de105d/bond-cni-plugin/0.log"
Jul 10 19:44:27.288584 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.288560    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2dpm8_2a5f5bf2-ff2e-49c2-b688-b1b298de105d/routeoverride-cni/0.log"
Jul 10 19:44:27.315170 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.315148    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2dpm8_2a5f5bf2-ff2e-49c2-b688-b1b298de105d/whereabouts-cni-bincopy/0.log"
Jul 10 19:44:27.339754 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.339725    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2dpm8_2a5f5bf2-ff2e-49c2-b688-b1b298de105d/whereabouts-cni/0.log"
Jul 10 19:44:27.755323 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.755296    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xbw6g_ddf22656-c128-49dc-80fb-94a983d5ebad/kube-multus/0.log"
Jul 10 19:44:27.879264 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.879228    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j7qn6_7b0bfc28-5d5c-42b3-aada-e8f37187da96/network-metrics-daemon/0.log"
Jul 10 19:44:27.896443 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:27.896420    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-j7qn6_7b0bfc28-5d5c-42b3-aada-e8f37187da96/kube-rbac-proxy/0.log"
Jul 10 19:44:28.670057 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.670023    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-controller/0.log"
Jul 10 19:44:28.689174 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.689149    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/0.log"
Jul 10 19:44:28.689322 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.689312    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovn-acl-logging/1.log"
Jul 10 19:44:28.719975 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.719955    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/kube-rbac-proxy-node/0.log"
Jul 10 19:44:28.747198 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.747176    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/kube-rbac-proxy-ovn-metrics/0.log"
Jul 10 19:44:28.769470 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.769444    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/northd/0.log"
Jul 10 19:44:28.791219 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.791198    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/nbdb/0.log"
Jul 10 19:44:28.814371 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.814337    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/sbdb/0.log"
Jul 10 19:44:28.895865 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:28.895833    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4l7jt_86ec85bc-771c-42e6-8c4b-6a31203c35c9/ovnkube-controller/0.log"
Jul 10 19:44:30.366415 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:30.366385    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-p7nlk_5a7bf1fa-bfba-4190-be81-f7bada15234e/network-check-target-container/0.log"
Jul 10 19:44:31.345588 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:31.345563    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8rgnm_6c45d7ef-4d75-46f9-8874-2e01a2a5945b/iptables-alerter/0.log"
Jul 10 19:44:32.054807 ip-10-0-140-163 kubenswrapper[2558]: I0710 19:44:32.054774    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xh9z9_cd859782-385f-4f5b-96cd-f10e405bfdbb/tuned/0.log"