Sep 17 20:47:28.757416 ip-10-0-143-254 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 20:47:28.757431 ip-10-0-143-254 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 20:47:28.757442 ip-10-0-143-254 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 20:47:28.757669 ip-10-0-143-254 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 20:47:38.901662 ip-10-0-143-254 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 20:47:38.901678 ip-10-0-143-254 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 36d52bd912e845e084746f840199ebab --
Sep 17 20:49:42.822819 ip-10-0-143-254 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 20:49:46.453507 ip-10-0-143-254 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:46.453507 ip-10-0-143-254 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 20:49:46.453507 ip-10-0-143-254 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:46.453507 ip-10-0-143-254 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 20:49:46.453507 ip-10-0-143-254 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 20:49:46.454454 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.453582    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 20:49:46.458443 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458425    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:46.458443 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458442    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458448    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458453    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458456    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458459    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458462    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458466    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458469    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458471    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458475    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458479    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458483    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458485    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458488    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458491    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458499    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458502    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458504    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458507    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:46.458512 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458510    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458512    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458515    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458518    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458521    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458524    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458526    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458529    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458532    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458534    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458539    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458542    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458545    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458548    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458550    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458553    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458556    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458558    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458561    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:46.459024 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458564    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458566    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458569    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458572    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458574    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458577    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458580    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458582    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458586    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458588    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458591    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458594    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458596    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458599    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458601    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458605    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458608    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458611    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458614    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458616    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:46.459555 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458619    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458622    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458624    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458627    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458629    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458633    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458635    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458638    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458641    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458643    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458646    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458648    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458652    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458654    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458657    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458659    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458662    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458664    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458667    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458669    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:46.460055 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458672    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458674    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458677    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458680    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458682    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458685    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458688    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458692    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458695    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458698    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458700    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458703    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458705    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458708    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458710    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458713    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458715    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458718    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458721    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.458723    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:46.460557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459469    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459477    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459480    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459483    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459486    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459489    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459492    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459496    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459498    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459501    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459505    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459509    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459512    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459515    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459518    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459521    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459524    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459527    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459529    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459533    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:46.461041 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459536    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459539    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459541    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459544    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459547    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459550    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459552    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459556    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459559    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459561    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459564    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459567    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459569    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459573    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459577    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459579    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459582    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459585    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459588    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:46.461557 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459591    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459594    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459597    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459599    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459602    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459604    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459607    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459611    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459613    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459617    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459621    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459624    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459628    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459631    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459633    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459636    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459639    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459641    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459644    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:46.462025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459647    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459650    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459652    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459655    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459657    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459660    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459662    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459666    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459669    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459671    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459674    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459677    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459679    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459682    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459685    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459688    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459690    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459693    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459695    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:46.462518 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459698    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459700    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459703    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459706    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459708    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459711    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459714    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459717    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459720    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459722    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459725    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459727    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459730    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459733    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459735    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459738    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459740    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459743    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459745    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459748    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:46.462990 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459750    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.459753    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460175    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460180    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460182    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460185    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460193    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460196    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460199    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460202    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460204    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460207    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460210    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460212    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460215    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460219    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460222    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460225    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460229    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:46.463502 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460233    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460235    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460238    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460241    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460243    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460246    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460248    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460251    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460253    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460256    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460258    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460261    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460263    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460266    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460268    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460271    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460274    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460276    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460279    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460281    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:46.464009 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460290    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460293    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460296    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460298    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460301    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460303    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460306    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460308    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460311    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460314    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460318    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460321    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460340    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460343    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460346    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460366    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460369    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460374    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460377    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:46.464527 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460380    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460383    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460386    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460389    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460392    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460394    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460397    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460399    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460402    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460404    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460407    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460411    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460413    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460416    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460424    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460427    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460429    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460432    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460434    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460437    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:46.464997 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460439    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460442    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460444    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460447    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460450    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460453    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460456    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460458    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460461    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460464    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460466    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460469    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460471    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460474    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460476    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460479    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460482    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460485    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460487    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460490    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:46.465488 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460492    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460495    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.460498    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460570    2586 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460604    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460614    2586 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460618    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460622    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460626    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460630    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460634    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460638    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460641    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460645    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460648    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460652    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460655    2586 flags.go:64] FLAG: --cgroup-root=""
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460659    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460661    2586 flags.go:64] FLAG: --client-ca-file=""
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460664    2586 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460667    2586 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460674    2586 flags.go:64] FLAG: --cluster-domain=""
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460677    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460680    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 20:49:46.465981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460683    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460687    2586 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460691    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460694    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460697    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460701    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460704    2586 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460707    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460712    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460715    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460718    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460722    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460725    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460728    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460731    2586 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460733    2586 flags.go:64] FLAG: --enable-server="true"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460737    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460742    2586 flags.go:64] FLAG: --event-burst="100"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460745    2586 flags.go:64] FLAG: --event-qps="50"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460748    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460751    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460754    2586 flags.go:64] FLAG: --eviction-hard=""
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460758    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460761    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460764    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 20:49:46.466588 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460767    2586 flags.go:64] FLAG: --eviction-soft=""
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460770    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460773    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460776    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460779    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460782    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460784    2586 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460787    2586 flags.go:64] FLAG: --feature-gates=""
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460791    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460794    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460797    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460800    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460803    2586 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460806    2586 flags.go:64] FLAG: --help="false"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460809    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460814    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460817    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460820    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460823    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460826    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460829    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460832    2586 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460834    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 20:49:46.467192 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460837    2586 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460841    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460844    2586 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460847    2586 flags.go:64] FLAG: --kube-reserved=""
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460850    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460853    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460856    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460859    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460862    2586 flags.go:64] FLAG: --lock-file=""
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460865    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460868    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460871    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460876    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460879    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460882    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460884    2586 flags.go:64] FLAG: --logging-format="text"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460887    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460891    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460894    2586 flags.go:64] FLAG: --manifest-url=""
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460896    2586 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460901    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460904    2586 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460908    2586 flags.go:64] FLAG: --max-pods="110"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460911    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460916    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 20:49:46.467836 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460919    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460922    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460925    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460928    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460931    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460938    2586 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460941    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460944    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460947    2586 flags.go:64] FLAG: --pod-cidr=""
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460956    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460961    2586 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460964    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460967    2586 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460970    2586 flags.go:64] FLAG: --port="10250"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460973    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460976    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cdf8d3faa94e55af"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460979    2586 flags.go:64] FLAG: --qos-reserved=""
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460982    2586 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460985    2586 flags.go:64] FLAG: --register-node="true"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460988    2586 flags.go:64] FLAG: --register-with-taints=""
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460992    2586 flags.go:64] FLAG: --registry-burst="10"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460995    2586 flags.go:64] FLAG: --registry-qps="5"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.460998    2586 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461001    2586 flags.go:64] FLAG: --reserved-memory=""
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461005    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 20:49:46.468528 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461008    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461011    2586 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461014    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461017    2586 flags.go:64] FLAG: --runonce="false"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461020    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461023    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461026    2586 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461030    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461033    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461035    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461039    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461042    2586 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461045    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461048    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461051    2586 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461054    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461057    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461066    2586 flags.go:64] FLAG: --system-cgroups=""
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461069    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461074    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461077    2586 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461082    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461085    2586 flags.go:64] FLAG: --tls-min-version=""
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461088    2586 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461091    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 20:49:46.469127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461094    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461097    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461100    2586 flags.go:64] FLAG: --v="2"
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461104    2586 flags.go:64] FLAG: --version="false"
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461108    2586 flags.go:64] FLAG: --vmodule=""
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461112    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.461115    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461239    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461242    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461245    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461250    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461253    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461257    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461260    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461264    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461267    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461269    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461272    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461275    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461278    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461281    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:46.469755 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461284    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461287    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461291    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461293    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461297    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461300    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461303    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461307    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461310    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461313    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461315    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461318    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461320    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461323    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461325    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461328    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461331    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461333    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461336    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461338    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:46.470334 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461341    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461344    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461346    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461364    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461367    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461371    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461374    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461377    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461380    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461382    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461385    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461388    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461390    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461393    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461395    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461398    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461401    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461412    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461415    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461420    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:46.471133 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461422    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461425    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461427    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461430    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461433    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461435    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461438    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461440    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461443    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461446    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461448    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461451    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461453    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461456    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461459    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461462    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461464    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461468    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461471    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:46.471743 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461474    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461476    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461479    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461481    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461484    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461487    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461489    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461492    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461494    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461497    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461499    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461507    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461511    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461514    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461517    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461519    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461522    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461525    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461527    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461530    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:46.472228 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461532    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461535    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461539    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461543    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461546    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.461550    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.462387    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.469331    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.469347    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.469383    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.469389    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469446    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469452    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469455    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469458    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:46.472764 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469461    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469464    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469467    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469471    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469474    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469477    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469480    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469483    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469486    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469489    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469491    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469494    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469497    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469499    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469502    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469505    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469508    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469510    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469513    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469516    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:46.473146 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469519    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469521    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469524    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469526    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469529    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469534    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469537    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469541    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469545    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469548    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469551    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469555    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469559    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469562    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469565    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469568    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469570    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469573    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469576    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:46.473652 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469579    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469581    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469584    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469587    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469590    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469593    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469596    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469598    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469601    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469604    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469606    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469609    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469612    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469614    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469617    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469620    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469622    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469625    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469628    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469632    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:46.474123 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469635    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469638    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469641    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469644    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469646    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469649    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469652    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469654    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469657    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469660    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469662    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469665    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469668    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469670    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469673    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469675    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469678    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469681    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469683    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469686    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:46.474668 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469688    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469691    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469694    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469696    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469699    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469701    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469704    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469707    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469709    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469712    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469715    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469720    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469724    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469728    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469731    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:46.475151 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469734    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.469739    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469855    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469860    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469863    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469866    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469869    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469872    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469874    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469877    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469880    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469882    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469885    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469887    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469890    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469893    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 20:49:46.475540 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469896    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469899    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469903    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469907    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469909    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469912    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469915    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469917    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469920    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469922    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469925    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469927    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469931    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469933    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469937    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469940    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469943    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469946    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469948    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469951    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 20:49:46.475947 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469954    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469956    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469959    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469961    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469964    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469966    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469969    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469972    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469974    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469977    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469980    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469982    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469985    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469987    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469990    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469993    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469995    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.469998    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470000    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 20:49:46.476641 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470004    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470006    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470009    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470011    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470014    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470017    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470019    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470022    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470025    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470028    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470032    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470036    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470038    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470041    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470045    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470047    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470050    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470053    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470056    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 20:49:46.477098 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470059    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470062    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470064    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470067    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470069    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470072    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470075    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470077    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470080    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470083    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470087    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470091    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470094    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470097    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470100    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470103    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470106    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470108    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470111    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470114    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 20:49:46.477594 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470116    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470119    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470161    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470201    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470206    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470211    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:46.470215    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.470224    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.471871    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.473753    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.474858    2586 server.go:1019] "Starting client certificate rotation"
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.474961    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:46.478077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.474988    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 20:49:46.478394 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.478109    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:46.480252 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.480236    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 20:49:46.484183 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.484166    2586 log.go:25] "Validated CRI v1 runtime API"
Sep 17 20:49:46.489447 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.489431    2586 log.go:25] "Validated CRI v1 image API"
Sep 17 20:49:46.490285 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.490269    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 20:49:46.492682 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.492655    2586 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e025243d-909a-4eab-a155-1722e3ca1ae6:/dev/nvme0n1p3 ea7ecfc9-1d1e-41cd-bf96-1e98b8ef7970:/dev/nvme0n1p4]
Sep 17 20:49:46.492743 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.492681    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 20:49:46.496818 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.496717    2586 manager.go:216] Machine: {Timestamp:2026-09-17 20:49:46.496244143 +0000 UTC m=+2.030183943 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102717 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23296bc4905353f1f3829283e6b191 SystemUUID:ec23296b-c490-5353-f1f3-829283e6b191 BootID:36d52bd9-12e8-45e0-8474-6f840199ebab Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e5:c5:38:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e5:c5:38:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:be:bb:de:14:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 20:49:46.496818 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.496814    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 20:49:46.496928 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.496891    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 20:49:46.498381 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.498338    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 20:49:46.498634 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.498379    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-254.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 20:49:46.499736 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.499291    2586 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 20:49:46.499736 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.499319    2586 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 20:49:46.499736 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.499331    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:46.499736 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.499348    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 20:49:46.499736 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.499458    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 20:49:46.503271 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503257    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:46.503401 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503391    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 20:49:46.503441 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503426    2586 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 20:49:46.503441 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503436    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 20:49:46.503495 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503450    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 20:49:46.503495 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503459    2586 kubelet.go:398] "Adding apiserver pod source"
Sep 17 20:49:46.503495 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503468    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 20:49:46.503833 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503813    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:46.503887 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.503835    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 20:49:46.505519 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.505502    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 20:49:46.512085 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.512066    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 20:49:46.512171 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.512104    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 20:49:46.512263 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.512240    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:46.513317 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513300    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 20:49:46.513317 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513320    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513327    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513335    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513346    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513374    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513394    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513402    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513410    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513415    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513424    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513429    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.513426    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.513426    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513452    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 20:49:46.513455 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.513460    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 20:49:46.516925 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.516901    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-254.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 20:49:46.517285 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.517271    2586 server.go:1268] "Started kubelet"
Sep 17 20:49:46.517373 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.517333    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 20:49:46.517467 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.517409    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 20:49:46.517516 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.517496    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 20:49:46.518294 ip-10-0-143-254 systemd[1]: Started Kubernetes Kubelet.
Sep 17 20:49:46.518916 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.518532    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 20:49:46.518916 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.518786    2586 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 20:49:46.521716 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.521682    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 20:49:46.524187 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.524170    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 20:49:46.524187 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.524187    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:46.524317 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.524238    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:46.524764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.524748    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 20:49:46.524842 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.524750    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 20:49:46.524842 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.524796    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 20:49:46.524945 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.524932    2586 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 20:49:46.524945 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.524943    2586 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 20:49:46.525517 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.525497    2586 factory.go:55] Registering systemd factory
Sep 17 20:49:46.525589 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.525520    2586 factory.go:223] Registration of the systemd container factory successfully
Sep 17 20:49:46.525744 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.525729    2586 factory.go:153] Registering CRI-O factory
Sep 17 20:49:46.525810 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.525747    2586 factory.go:223] Registration of the crio container factory successfully
Sep 17 20:49:46.525862 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.525817    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 20:49:46.525862 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.525832    2586 factory.go:103] Registering Raw factory
Sep 17 20:49:46.525862 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.525847    2586 manager.go:1196] Started watching for new ooms in manager
Sep 17 20:49:46.526606 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.526591    2586 manager.go:349] Starting recovery of all containers
Sep 17 20:49:46.526691 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.526617    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 20:49:46.526752 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.526701    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 20:49:46.557554 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.557532    2586 manager.go:354] Recovery completed
Sep 17 20:49:46.561980 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.561967    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:46.564607 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.564588    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:46.564666 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.564622    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:46.564666 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.564634    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:46.565213 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.565199    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 20:49:46.565213 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.565213    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 20:49:46.565293 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.565225    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:46.565293 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.565228    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 20:49:46.565986 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.565965    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:46.566077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.565999    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:46.566077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.566012    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:46.566573 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.566560    2586 policy_none.go:49] "None policy: Start"
Sep 17 20:49:46.566627 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.566576    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 20:49:46.566627 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.566587    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 20:49:46.566826 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.565857    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-254.ec2.internal.18d636d419a01e38  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-254.ec2.internal,UID:ip-10-0-143-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-254.ec2.internal,},FirstTimestamp:2026-09-17 20:49:46.5172414 +0000 UTC m=+2.051181202,LastTimestamp:2026-09-17 20:49:46.5172414 +0000 UTC m=+2.051181202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-254.ec2.internal,}"
Sep 17 20:49:46.567529 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.567517    2586 policy_none.go:47] "Start"
Sep 17 20:49:46.568454 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.568437    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hrgjm"
Sep 17 20:49:46.576014 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.575811    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-254.ec2.internal.18d636d41c72dc18  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-254.ec2.internal,UID:ip-10-0-143-254.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-254.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-254.ec2.internal,},FirstTimestamp:2026-09-17 20:49:46.564607 +0000 UTC m=+2.098546799,LastTimestamp:2026-09-17 20:49:46.564607 +0000 UTC m=+2.098546799,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-254.ec2.internal,}"
Sep 17 20:49:46.576957 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.576941    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hrgjm"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.598934    2586 manager.go:337] "Starting Device Plugin manager"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.598993    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.599008    2586 server.go:87] "Starting device plugin registration server"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.599259    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.599287    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.599478    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.599569    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 20:49:46.600083 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.599580    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 20:49:46.600573 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.600156    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 20:49:46.600573 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.600257    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:46.664246 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.664210    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 20:49:46.665479 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.665464    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 20:49:46.665540 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.665518    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 20:49:46.665579 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.665550    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 20:49:46.665632 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.665602    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 20:49:46.668137 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.668115    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:46.700278 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.700235    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:46.701985 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.701968    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:46.702087 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.701998    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:46.702087 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.702012    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:46.702087 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.702040    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.711719 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.711670    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.711719 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.711696    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-254.ec2.internal\": node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:46.746580 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.746545    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:46.767327 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.766713    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal"]
Sep 17 20:49:46.767481 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.767405    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:46.772598 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.772583    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:46.772683 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.772613    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:46.772683 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.772624    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:46.773853 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.773839    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:46.773997 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.773982    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.774059 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.774015    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:46.775187 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.775174    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:46.775187 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.775178    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:46.775323 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.775201    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:46.775323 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.775202    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:46.775323 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.775226    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:46.775323 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.775211    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:46.776334 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.776319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.776397 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.776378    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:46.777127 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.777108    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:46.777241 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.777135    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:46.777241 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.777151    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:46.790917 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.790898    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-254.ec2.internal\" not found" node="ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.795469 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.795452    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-254.ec2.internal\" not found" node="ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.826600 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.826572    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be62ecf7e2d5494a2c86866674c5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal\" (UID: \"7be62ecf7e2d5494a2c86866674c5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.826600 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.826605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be62ecf7e2d5494a2c86866674c5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal\" (UID: \"7be62ecf7e2d5494a2c86866674c5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.826800 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.826632    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d17c601b9e0bec6e3d509315a4aa586-config\") pod \"kube-apiserver-proxy-ip-10-0-143-254.ec2.internal\" (UID: \"0d17c601b9e0bec6e3d509315a4aa586\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.847669 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.847645    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:46.927725 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.927692    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be62ecf7e2d5494a2c86866674c5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal\" (UID: \"7be62ecf7e2d5494a2c86866674c5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.927839 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.927735    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be62ecf7e2d5494a2c86866674c5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal\" (UID: \"7be62ecf7e2d5494a2c86866674c5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.927839 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.927762    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d17c601b9e0bec6e3d509315a4aa586-config\") pod \"kube-apiserver-proxy-ip-10-0-143-254.ec2.internal\" (UID: \"0d17c601b9e0bec6e3d509315a4aa586\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.927839 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.927782    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be62ecf7e2d5494a2c86866674c5e01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal\" (UID: \"7be62ecf7e2d5494a2c86866674c5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.927839 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.927820    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be62ecf7e2d5494a2c86866674c5e01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal\" (UID: \"7be62ecf7e2d5494a2c86866674c5e01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.927982 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:46.927820    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0d17c601b9e0bec6e3d509315a4aa586-config\") pod \"kube-apiserver-proxy-ip-10-0-143-254.ec2.internal\" (UID: \"0d17c601b9e0bec6e3d509315a4aa586\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:46.947740 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:46.947701    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.048477 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.048394    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.094585 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.094557    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:47.099519 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.099037    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:47.149478 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.149443    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.250032 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.250004    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.350588 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.350512    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.451104 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.451057    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.474244 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.474215    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 20:49:47.474670 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.474391    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 20:49:47.520302 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.520271    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:47.551587 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.551557    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.581764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.581726    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 20:44:46 +0000 UTC" deadline="2028-06-03 22:21:12.561930522 +0000 UTC"
Sep 17 20:49:47.581764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.581758    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15001h31m24.980175808s"
Sep 17 20:49:47.636338 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.636260    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 20:49:47.638563 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.638545    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 20:49:47.638665 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.638576    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 20:49:47.638665 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.638587    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeHasSufficientPID"
Sep 17 20:49:47.652329 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.652307    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.717262 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:47.717230    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d17c601b9e0bec6e3d509315a4aa586.slice/crio-e880b220c740c529567768801397616844c8ce5665f140fa080bb3ca14ab7880 WatchSource:0}: Error finding container e880b220c740c529567768801397616844c8ce5665f140fa080bb3ca14ab7880: Status 404 returned error can't find the container with id e880b220c740c529567768801397616844c8ce5665f140fa080bb3ca14ab7880
Sep 17 20:49:47.718005 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.717985    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:47.725604 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.725586    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:49:47.726273 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:47.726247    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be62ecf7e2d5494a2c86866674c5e01.slice/crio-af26cc09ef247831bfce6e1ef8185495174e98359c3f06f9045f455dadba106c WatchSource:0}: Error finding container af26cc09ef247831bfce6e1ef8185495174e98359c3f06f9045f455dadba106c: Status 404 returned error can't find the container with id af26cc09ef247831bfce6e1ef8185495174e98359c3f06f9045f455dadba106c
Sep 17 20:49:47.753423 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:47.753393    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-254.ec2.internal\" not found"
Sep 17 20:49:47.758610 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.758593    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:47.824467 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.824410    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:47.840001 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.839976    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:47.841235 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.841221    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal"
Sep 17 20:49:47.849638 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:47.849620    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 20:49:48.382068 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.382037    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 20:49:48.505641 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.505607    2586 apiserver.go:52] "Watching apiserver"
Sep 17 20:49:48.512271 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.512242    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 20:49:48.512747 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.512720    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal","openshift-multus/multus-additional-cni-plugins-d85tw","openshift-multus/multus-p8cpg","openshift-multus/network-metrics-daemon-2rvtb","kube-system/konnectivity-agent-rrt59","openshift-cluster-node-tuning-operator/tuned-fszzw","openshift-dns/node-resolver-wlqs4","openshift-image-registry/node-ca-7c4kq","openshift-network-diagnostics/network-check-target-v57r9","openshift-network-operator/iptables-alerter-wjnf2","openshift-ovn-kubernetes/ovnkube-node-hg45d","kube-system/global-pull-secret-syncer-x4x76","kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"]
Sep 17 20:49:48.514306 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.514217    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.515855 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.515721    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.516842 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.516655    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.516929 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.516904    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 20:49:48.516983 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.516956    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fsgqd\""
Sep 17 20:49:48.517407 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.517391    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 20:49:48.518389 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.517949    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:48.518389 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.518065    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:49:48.518389 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.518154    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-57w4k\""
Sep 17 20:49:48.519173 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.519152    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.519277 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.519237    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 20:49:48.519368 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.519274    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 20:49:48.519655 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.519635    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 20:49:48.519778 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.519753    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 20:49:48.519934 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.519921    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cqp4q\""
Sep 17 20:49:48.520006 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.519954    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 20:49:48.520952 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.520824    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.521856 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.521661    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 20:49:48.521856 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.521811    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4q72w\""
Sep 17 20:49:48.522134 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.522026    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.522219 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.522157    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 20:49:48.522219 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.522180    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 20:49:48.523165 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.523021    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:48.523235 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.523219    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kcvn7\""
Sep 17 20:49:48.523306 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.523240    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:48.523719 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.523671    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.524232 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.524213    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v7f42\""
Sep 17 20:49:48.524596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.524431    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 20:49:48.524596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.524460    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 20:49:48.524772 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.524731    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 20:49:48.525197 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.525177    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.526001 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.525864    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 20:49:48.526721 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.526090    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 20:49:48.526721 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.526267    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 20:49:48.526721 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.526532    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-52rmj\""
Sep 17 20:49:48.526721 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.526538    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 20:49:48.526981 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.526730    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.528397 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.527779    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.528818 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.528791    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:49:48.531404 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.529608    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 20:49:48.531561 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.531544    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:48.532068 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530315    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 20:49:48.532068 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.532010    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:49:48.532068 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530498    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 20:49:48.533023 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530413    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qfkhw\""
Sep 17 20:49:48.533111 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530713    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 20:49:48.533168 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530719    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 20:49:48.533326 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530859    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 20:49:48.533477 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530882    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 20:49:48.533477 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530927    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 20:49:48.533586 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.529643    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cb2gq\""
Sep 17 20:49:48.533641 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.530565    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536563    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-ovn\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536598    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536624    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536647    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-cni-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536670    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eac2d386-b829-49d2-ab71-42201ebfcf07-konnectivity-ca\") pod \"konnectivity-agent-rrt59\" (UID: \"eac2d386-b829-49d2-ab71-42201ebfcf07\") " pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536692    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-kubernetes\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536713    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536734    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-env-overrides\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536756    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-netns\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536780    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-kubelet\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.536847 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536801    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-multus-certs\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536855    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn65n\" (UniqueName: \"kubernetes.io/projected/0efd83dd-81e7-487a-b68e-2b9197c5f655-kube-api-access-vn65n\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536894    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-run\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536922    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmlf\" (UniqueName: \"kubernetes.io/projected/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-kube-api-access-mjmlf\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536947    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-ovnkube-config\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536968    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-lib-modules\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536983    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2f69\" (UniqueName: \"kubernetes.io/projected/459fe0d3-2504-4964-b4ef-4161f8229f68-kube-api-access-h2f69\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.536997    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eac2d386-b829-49d2-ab71-42201ebfcf07-agent-certs\") pod \"konnectivity-agent-rrt59\" (UID: \"eac2d386-b829-49d2-ab71-42201ebfcf07\") " pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537014    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysctl-d\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537035    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-systemd\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537092    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-sys\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537114    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537130    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-system-cni-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537165    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-tuned\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537187    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-registration-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537225    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-socket-dir-parent\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.537387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537240    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-daemon-config\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537291    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-host\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537321    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-sys-fs\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537383    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqkq\" (UniqueName: \"kubernetes.io/projected/8f3930b3-92c5-4083-94fe-1881c5969abd-kube-api-access-9sqkq\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537405    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-systemd\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537427    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/818325ca-b623-4f67-8664-9e8d89e50ddc-ovn-node-metrics-cert\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537451    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-cni-binary-copy\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5174cf12-96bf-4526-b5ee-66dcc04371c0-cni-binary-copy\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537498    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-etc-kubernetes\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537521    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-socket-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537547    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-var-lib-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537574    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-log-socket\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537627    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-cnibin\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537650    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-device-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537684    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538092 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537726    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzmvv\" (UniqueName: \"kubernetes.io/projected/818325ca-b623-4f67-8664-9e8d89e50ddc-kube-api-access-hzmvv\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537752    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-cnibin\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537780    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-conf-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537803    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysctl-conf\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537828    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsrr\" (UniqueName: \"kubernetes.io/projected/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-kube-api-access-mhsrr\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537883    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-modprobe-d\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537904    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/459fe0d3-2504-4964-b4ef-4161f8229f68-tmp\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537926    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gc44\" (UniqueName: \"kubernetes.io/projected/078730b6-ff30-4d6b-9146-451243e73b49-kube-api-access-4gc44\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537948    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-tmp-dir\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.537979    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f3930b3-92c5-4083-94fe-1881c5969abd-serviceca\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538000    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-systemd-units\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538029    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlhx\" (UniqueName: \"kubernetes.io/projected/5200bd98-2df0-453a-9ea0-251c997ede89-kube-api-access-8jlhx\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538066    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-cni-bin\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538103    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-var-lib-kubelet\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538143    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f3930b3-92c5-4083-94fe-1881c5969abd-host\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.538824 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538194    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-slash\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538276    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-run-netns\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538305    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-node-log\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538445    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-cni-bin\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538472    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4d7w\" (UniqueName: \"kubernetes.io/projected/5174cf12-96bf-4526-b5ee-66dcc04371c0-kube-api-access-l4d7w\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538494    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-system-cni-dir\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538520    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-k8s-cni-cncf-io\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538580    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysconfig\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538609    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/078730b6-ff30-4d6b-9146-451243e73b49-iptables-alerter-script\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538634    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-etc-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538666    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-cni-netd\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538764    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-ovnkube-script-lib\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538828    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538854    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-cni-multus\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538887    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/078730b6-ff30-4d6b-9146-451243e73b49-host-slash\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538911    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-hosts-file\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.539543 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538974    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.540208 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.538997    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-hostroot\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.540208 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.539087    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-kubelet\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.542398 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.542344    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 20:49:48.562918 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.562890    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4wh8w"
Sep 17 20:49:48.572389 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.572334    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4wh8w"
Sep 17 20:49:48.626373 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.626336    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 20:49:48.640474 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640348    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysctl-conf\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.640474 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640417    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsrr\" (UniqueName: \"kubernetes.io/projected/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-kube-api-access-mhsrr\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.640753 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640619    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.640753 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640647    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysctl-conf\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.640753 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640671    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32ea5724-e1d9-4592-8b7c-32e9d08b5388-dbus\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.640753 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640721    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640794    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-modprobe-d\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/459fe0d3-2504-4964-b4ef-4161f8229f68-tmp\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640848    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gc44\" (UniqueName: \"kubernetes.io/projected/078730b6-ff30-4d6b-9146-451243e73b49-kube-api-access-4gc44\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640876    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-tmp-dir\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640902    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f3930b3-92c5-4083-94fe-1881c5969abd-serviceca\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-systemd-units\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640956    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlhx\" (UniqueName: \"kubernetes.io/projected/5200bd98-2df0-453a-9ea0-251c997ede89-kube-api-access-8jlhx\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640965    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-modprobe-d\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.640983    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-cni-bin\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.641007 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-var-lib-kubelet\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641033    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f3930b3-92c5-4083-94fe-1881c5969abd-host\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641055    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-slash\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641079    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-run-netns\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641101    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-node-log\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641124    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-cni-bin\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641163    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4d7w\" (UniqueName: \"kubernetes.io/projected/5174cf12-96bf-4526-b5ee-66dcc04371c0-kube-api-access-l4d7w\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641189    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-system-cni-dir\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-k8s-cni-cncf-io\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysconfig\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641268    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/078730b6-ff30-4d6b-9146-451243e73b49-iptables-alerter-script\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641306    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-etc-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641334    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-cni-netd\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641375    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-ovnkube-script-lib\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641349    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641431    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-cni-multus\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641456    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/078730b6-ff30-4d6b-9146-451243e73b49-host-slash\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.641494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641479    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-hosts-file\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641503    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641527    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-hostroot\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641550    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-kubelet\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641581    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-ovn\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641614    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641664    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-cni-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641688    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eac2d386-b829-49d2-ab71-42201ebfcf07-konnectivity-ca\") pod \"konnectivity-agent-rrt59\" (UID: \"eac2d386-b829-49d2-ab71-42201ebfcf07\") " pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641780    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysconfig\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641836    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8f3930b3-92c5-4083-94fe-1881c5969abd-host\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641839    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-slash\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641865    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-run-netns\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-systemd-units\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641947    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-cni-bin\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.641971    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-node-log\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642144    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-system-cni-dir\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.642298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642194    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-kubernetes\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642221    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642247    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-env-overrides\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-netns\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642289    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eac2d386-b829-49d2-ab71-42201ebfcf07-konnectivity-ca\") pod \"konnectivity-agent-rrt59\" (UID: \"eac2d386-b829-49d2-ab71-42201ebfcf07\") " pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642317    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-kubelet\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642379    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-var-lib-kubelet\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-k8s-cni-cncf-io\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642444    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-run-ovn-kubernetes\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-tmp-dir\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642505    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-etc-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642505    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-kubelet\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642518    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-hostroot\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642548    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-ovn\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642554    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-host-cni-netd\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642578    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-cni-bin\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642661    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-etc-selinux\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642698    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-cni-multus\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642734    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/078730b6-ff30-4d6b-9146-451243e73b49-host-slash\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642292    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-var-lib-kubelet\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642779    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-hosts-file\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642787    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-multus-certs\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642825    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn65n\" (UniqueName: \"kubernetes.io/projected/0efd83dd-81e7-487a-b68e-2b9197c5f655-kube-api-access-vn65n\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642850    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-run\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642888    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmlf\" (UniqueName: \"kubernetes.io/projected/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-kube-api-access-mjmlf\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642918    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-ovnkube-config\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-lib-modules\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642952    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/078730b6-ff30-4d6b-9146-451243e73b49-iptables-alerter-script\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.642990    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-cni-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643017    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2f69\" (UniqueName: \"kubernetes.io/projected/459fe0d3-2504-4964-b4ef-4161f8229f68-kube-api-access-h2f69\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643050    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643060    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-kubernetes\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643081    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eac2d386-b829-49d2-ab71-42201ebfcf07-agent-certs\") pod \"konnectivity-agent-rrt59\" (UID: \"eac2d386-b829-49d2-ab71-42201ebfcf07\") " pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysctl-d\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-systemd\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.643940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643172    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643290    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643318    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8f3930b3-92c5-4083-94fe-1881c5969abd-serviceca\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643335    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-run\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643388    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-multus-certs\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643132    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643460    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-lib-modules\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-sysctl-d\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643615    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-ovnkube-config\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643629    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-sys\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643617    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-host-run-netns\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643649    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-ovnkube-script-lib\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643661    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643694    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32ea5724-e1d9-4592-8b7c-32e9d08b5388-kubelet-config\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643711    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-sys\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643717    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-systemd\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643756    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-system-cni-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643771    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.644801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643790    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-tuned\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643834    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-system-cni-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-registration-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643932    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-registration-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-socket-dir-parent\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.643974    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-daemon-config\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644000    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-host\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644026    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/818325ca-b623-4f67-8664-9e8d89e50ddc-env-overrides\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644044    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-sys-fs\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644073    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqkq\" (UniqueName: \"kubernetes.io/projected/8f3930b3-92c5-4083-94fe-1881c5969abd-kube-api-access-9sqkq\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644084    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-socket-dir-parent\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644131    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/459fe0d3-2504-4964-b4ef-4161f8229f68-host\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644138    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-sys-fs\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644172    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-systemd\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/818325ca-b623-4f67-8664-9e8d89e50ddc-ovn-node-metrics-cert\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644243    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-systemd\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644298    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-cni-binary-copy\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.646084 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644343    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5174cf12-96bf-4526-b5ee-66dcc04371c0-cni-binary-copy\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644402    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-etc-kubernetes\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644433    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-socket-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644461    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-var-lib-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644488    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-log-socket\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644515    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644528    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-etc-kubernetes\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644545    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-cnibin\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644571    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-device-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644606    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-socket-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644623    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzmvv\" (UniqueName: \"kubernetes.io/projected/818325ca-b623-4f67-8664-9e8d89e50ddc-kube-api-access-hzmvv\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644645    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-daemon-config\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644696    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-cnibin\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.644700    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644715    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-device-dir\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644740    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-cnibin\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644651    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5200bd98-2df0-453a-9ea0-251c997ede89-cnibin\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.647055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644760    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-run-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.644792    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:49.144750789 +0000 UTC m=+4.678690579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644804    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-var-lib-openvswitch\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644810    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5200bd98-2df0-453a-9ea0-251c997ede89-cni-binary-copy\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644831    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/818325ca-b623-4f67-8664-9e8d89e50ddc-log-socket\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644873    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-conf-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.644947    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5174cf12-96bf-4526-b5ee-66dcc04371c0-multus-conf-dir\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.645028    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5174cf12-96bf-4526-b5ee-66dcc04371c0-cni-binary-copy\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.645800    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/459fe0d3-2504-4964-b4ef-4161f8229f68-tmp\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.647116    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/818325ca-b623-4f67-8664-9e8d89e50ddc-ovn-node-metrics-cert\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.647311    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/459fe0d3-2504-4964-b4ef-4161f8229f68-etc-tuned\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.647966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.647704    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eac2d386-b829-49d2-ab71-42201ebfcf07-agent-certs\") pod \"konnectivity-agent-rrt59\" (UID: \"eac2d386-b829-49d2-ab71-42201ebfcf07\") " pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.657247 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.657212    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsrr\" (UniqueName: \"kubernetes.io/projected/a2c6d687-7ce5-41d2-beea-c8e7833fc4e0-kube-api-access-mhsrr\") pod \"node-resolver-wlqs4\" (UID: \"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0\") " pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.657481 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.657212    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gc44\" (UniqueName: \"kubernetes.io/projected/078730b6-ff30-4d6b-9146-451243e73b49-kube-api-access-4gc44\") pod \"iptables-alerter-wjnf2\" (UID: \"078730b6-ff30-4d6b-9146-451243e73b49\") " pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.662286 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.662256    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqkq\" (UniqueName: \"kubernetes.io/projected/8f3930b3-92c5-4083-94fe-1881c5969abd-kube-api-access-9sqkq\") pod \"node-ca-7c4kq\" (UID: \"8f3930b3-92c5-4083-94fe-1881c5969abd\") " pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.663906 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.663882    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn65n\" (UniqueName: \"kubernetes.io/projected/0efd83dd-81e7-487a-b68e-2b9197c5f655-kube-api-access-vn65n\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:48.664023 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.663939    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzmvv\" (UniqueName: \"kubernetes.io/projected/818325ca-b623-4f67-8664-9e8d89e50ddc-kube-api-access-hzmvv\") pod \"ovnkube-node-hg45d\" (UID: \"818325ca-b623-4f67-8664-9e8d89e50ddc\") " pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:48.664900 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.664880    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2f69\" (UniqueName: \"kubernetes.io/projected/459fe0d3-2504-4964-b4ef-4161f8229f68-kube-api-access-h2f69\") pod \"tuned-fszzw\" (UID: \"459fe0d3-2504-4964-b4ef-4161f8229f68\") " pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.665168 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.665028    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlhx\" (UniqueName: \"kubernetes.io/projected/5200bd98-2df0-453a-9ea0-251c997ede89-kube-api-access-8jlhx\") pod \"multus-additional-cni-plugins-d85tw\" (UID: \"5200bd98-2df0-453a-9ea0-251c997ede89\") " pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.665395 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.665377    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4d7w\" (UniqueName: \"kubernetes.io/projected/5174cf12-96bf-4526-b5ee-66dcc04371c0-kube-api-access-l4d7w\") pod \"multus-p8cpg\" (UID: \"5174cf12-96bf-4526-b5ee-66dcc04371c0\") " pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.666973 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.666951    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmlf\" (UniqueName: \"kubernetes.io/projected/c4292a64-fe58-4798-b5a2-37dbcbeae4bc-kube-api-access-mjmlf\") pod \"aws-ebs-csi-driver-node-r74s9\" (UID: \"c4292a64-fe58-4798-b5a2-37dbcbeae4bc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.673943 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.673854    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal" event={"ID":"7be62ecf7e2d5494a2c86866674c5e01","Type":"ContainerStarted","Data":"af26cc09ef247831bfce6e1ef8185495174e98359c3f06f9045f455dadba106c"}
Sep 17 20:49:48.675214 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.675182    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal" event={"ID":"0d17c601b9e0bec6e3d509315a4aa586","Type":"ContainerStarted","Data":"e880b220c740c529567768801397616844c8ce5665f140fa080bb3ca14ab7880"}
Sep 17 20:49:48.745793 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.745756    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32ea5724-e1d9-4592-8b7c-32e9d08b5388-kubelet-config\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.745984 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.745835    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32ea5724-e1d9-4592-8b7c-32e9d08b5388-dbus\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.745984 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.745857    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/32ea5724-e1d9-4592-8b7c-32e9d08b5388-kubelet-config\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.745984 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.745862    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.745984 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.745950    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:48.746171 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.745940    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/32ea5724-e1d9-4592-8b7c-32e9d08b5388-dbus\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:48.746171 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.746012    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret podName:32ea5724-e1d9-4592-8b7c-32e9d08b5388 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:49.245995016 +0000 UTC m=+4.779934812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret") pod "global-pull-secret-syncer-x4x76" (UID: "32ea5724-e1d9-4592-8b7c-32e9d08b5388") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:48.746171 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.746051    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:48.752423 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.752401    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:48.752423 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.752422    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:48.752607 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.752436    2586 projected.go:196] Error preparing data for projected volume kube-api-access-nxnxn for pod openshift-network-diagnostics/network-check-target-v57r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:48.752607 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:48.752500    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn podName:be6cd316-29b1-4d56-9ae4-f9688eba78ec nodeName:}" failed. No retries permitted until 2026-09-17 20:49:49.252486186 +0000 UTC m=+4.786425973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnxn" (UniqueName: "kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn") pod "network-check-target-v57r9" (UID: "be6cd316-29b1-4d56-9ae4-f9688eba78ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:48.827072 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.827034    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wlqs4"
Sep 17 20:49:48.840894 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.840860    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:49:48.848804 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.848770    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d85tw"
Sep 17 20:49:48.862586 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.862556    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p8cpg"
Sep 17 20:49:48.870519 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.870493    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fszzw"
Sep 17 20:49:48.876178 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.876147    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9"
Sep 17 20:49:48.882686 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.882660    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7c4kq"
Sep 17 20:49:48.891399 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.891327    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wjnf2"
Sep 17 20:49:48.895969 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:48.895953    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:49:49.149316 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.149234    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:49.149491 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.149386    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:49.149491 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.149447    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:50.1494321 +0000 UTC m=+5.683371900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:49.249756 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.249727    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:49.249943 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.249918    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:49.250022 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.250009    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret podName:32ea5724-e1d9-4592-8b7c-32e9d08b5388 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:50.249987482 +0000 UTC m=+5.783927271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret") pod "global-pull-secret-syncer-x4x76" (UID: "32ea5724-e1d9-4592-8b7c-32e9d08b5388") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:49.350531 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.350495    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:49.350705 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.350642    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:49.350705 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.350668    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:49.350705 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.350682    2586 projected.go:196] Error preparing data for projected volume kube-api-access-nxnxn for pod openshift-network-diagnostics/network-check-target-v57r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:49.350827 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.350737    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn podName:be6cd316-29b1-4d56-9ae4-f9688eba78ec nodeName:}" failed. No retries permitted until 2026-09-17 20:49:50.350722382 +0000 UTC m=+5.884662168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnxn" (UniqueName: "kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn") pod "network-check-target-v57r9" (UID: "be6cd316-29b1-4d56-9ae4-f9688eba78ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:49.561347 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:49.561320    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3930b3_92c5_4083_94fe_1881c5969abd.slice/crio-fcc986c224fc2713553cfc044032d913f1a53447ccc83e3f4f751c39b2aa0401 WatchSource:0}: Error finding container fcc986c224fc2713553cfc044032d913f1a53447ccc83e3f4f751c39b2aa0401: Status 404 returned error can't find the container with id fcc986c224fc2713553cfc044032d913f1a53447ccc83e3f4f751c39b2aa0401
Sep 17 20:49:49.563328 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:49.562901    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac2d386_b829_49d2_ab71_42201ebfcf07.slice/crio-fade2090dd4292a1bb9d6b7180ac5f088787562b97dc59e6ee95cdcca4fc3ddc WatchSource:0}: Error finding container fade2090dd4292a1bb9d6b7180ac5f088787562b97dc59e6ee95cdcca4fc3ddc: Status 404 returned error can't find the container with id fade2090dd4292a1bb9d6b7180ac5f088787562b97dc59e6ee95cdcca4fc3ddc
Sep 17 20:49:49.563793 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:49.563768    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4292a64_fe58_4798_b5a2_37dbcbeae4bc.slice/crio-4c2d392a3bab562955ebad5737723fe62ca959a81db1bd0ff8ae0e41973554df WatchSource:0}: Error finding container 4c2d392a3bab562955ebad5737723fe62ca959a81db1bd0ff8ae0e41973554df: Status 404 returned error can't find the container with id 4c2d392a3bab562955ebad5737723fe62ca959a81db1bd0ff8ae0e41973554df
Sep 17 20:49:49.564799 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:49.564624    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078730b6_ff30_4d6b_9146_451243e73b49.slice/crio-cc2846e1800f21400b44d8db7f236d2415cf7ff11674a5b1d00436c0834bff92 WatchSource:0}: Error finding container cc2846e1800f21400b44d8db7f236d2415cf7ff11674a5b1d00436c0834bff92: Status 404 returned error can't find the container with id cc2846e1800f21400b44d8db7f236d2415cf7ff11674a5b1d00436c0834bff92
Sep 17 20:49:49.565455 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:49.565434    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459fe0d3_2504_4964_b4ef_4161f8229f68.slice/crio-a6381d3917fddc2c3e371a265a3b1ed1f4518400cc18dd7ac73fef397c37f741 WatchSource:0}: Error finding container a6381d3917fddc2c3e371a265a3b1ed1f4518400cc18dd7ac73fef397c37f741: Status 404 returned error can't find the container with id a6381d3917fddc2c3e371a265a3b1ed1f4518400cc18dd7ac73fef397c37f741
Sep 17 20:49:49.569794 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:49.569717    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c6d687_7ce5_41d2_beea_c8e7833fc4e0.slice/crio-b5f0b6e9ce3e80b245c1672109f8f7b2e97f3eabd126e443640b894e027f4b51 WatchSource:0}: Error finding container b5f0b6e9ce3e80b245c1672109f8f7b2e97f3eabd126e443640b894e027f4b51: Status 404 returned error can't find the container with id b5f0b6e9ce3e80b245c1672109f8f7b2e97f3eabd126e443640b894e027f4b51
Sep 17 20:49:49.574049 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.574017    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:48 +0000 UTC" deadline="2028-05-19 12:07:11.063355583 +0000 UTC"
Sep 17 20:49:49.574049 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.574050    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14631h17m21.489309848s"
Sep 17 20:49:49.602553 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:49:49.602520    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5174cf12_96bf_4526_b5ee_66dcc04371c0.slice/crio-9016a647044619dbc6171dc1ffc340737f61653e8ba8800203734a3609ddf52c WatchSource:0}: Error finding container 9016a647044619dbc6171dc1ffc340737f61653e8ba8800203734a3609ddf52c: Status 404 returned error can't find the container with id 9016a647044619dbc6171dc1ffc340737f61653e8ba8800203734a3609ddf52c
Sep 17 20:49:49.665730 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.665705    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:49.665909 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:49.665802    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:49:49.680349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.680320    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjnf2" event={"ID":"078730b6-ff30-4d6b-9146-451243e73b49","Type":"ContainerStarted","Data":"cc2846e1800f21400b44d8db7f236d2415cf7ff11674a5b1d00436c0834bff92"}
Sep 17 20:49:49.684126 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.684091    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9" event={"ID":"c4292a64-fe58-4798-b5a2-37dbcbeae4bc","Type":"ContainerStarted","Data":"4c2d392a3bab562955ebad5737723fe62ca959a81db1bd0ff8ae0e41973554df"}
Sep 17 20:49:49.687372 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.687333    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7c4kq" event={"ID":"8f3930b3-92c5-4083-94fe-1881c5969abd","Type":"ContainerStarted","Data":"fcc986c224fc2713553cfc044032d913f1a53447ccc83e3f4f751c39b2aa0401"}
Sep 17 20:49:49.690065 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.690039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal" event={"ID":"0d17c601b9e0bec6e3d509315a4aa586","Type":"ContainerStarted","Data":"e6c52eda2ba8909ecb2553dacc055716667bc181be4d3e68216de47ae7f0bea7"}
Sep 17 20:49:49.695900 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.695873    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8cpg" event={"ID":"5174cf12-96bf-4526-b5ee-66dcc04371c0","Type":"ContainerStarted","Data":"9016a647044619dbc6171dc1ffc340737f61653e8ba8800203734a3609ddf52c"}
Sep 17 20:49:49.697621 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.697599    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerStarted","Data":"723076b32786dc002cea2734bf77d03defa7c0abb303b93dbc678bd1ee7519ba"}
Sep 17 20:49:49.699720 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.699698    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"ca2c54172c379ce95f67ffd915165ac2ee854c8d45b695516e468be517f18a05"}
Sep 17 20:49:49.702669 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.702650    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wlqs4" event={"ID":"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0","Type":"ContainerStarted","Data":"b5f0b6e9ce3e80b245c1672109f8f7b2e97f3eabd126e443640b894e027f4b51"}
Sep 17 20:49:49.703668 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.703649    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fszzw" event={"ID":"459fe0d3-2504-4964-b4ef-4161f8229f68","Type":"ContainerStarted","Data":"a6381d3917fddc2c3e371a265a3b1ed1f4518400cc18dd7ac73fef397c37f741"}
Sep 17 20:49:49.705799 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:49.705744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rrt59" event={"ID":"eac2d386-b829-49d2-ab71-42201ebfcf07","Type":"ContainerStarted","Data":"fade2090dd4292a1bb9d6b7180ac5f088787562b97dc59e6ee95cdcca4fc3ddc"}
Sep 17 20:49:50.157232 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.156873    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:50.157232 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.157139    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:50.157232 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.157219    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:52.157198731 +0000 UTC m=+7.691138524 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:50.258989 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.258402    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:50.258989 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.258545    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:50.258989 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.258610    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret podName:32ea5724-e1d9-4592-8b7c-32e9d08b5388 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:52.2585898 +0000 UTC m=+7.792529589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret") pod "global-pull-secret-syncer-x4x76" (UID: "32ea5724-e1d9-4592-8b7c-32e9d08b5388") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:50.358994 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.358952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:50.359216 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.359196    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:50.359303 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.359224    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:50.359303 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.359239    2586 projected.go:196] Error preparing data for projected volume kube-api-access-nxnxn for pod openshift-network-diagnostics/network-check-target-v57r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:50.359440 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.359327    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn podName:be6cd316-29b1-4d56-9ae4-f9688eba78ec nodeName:}" failed. No retries permitted until 2026-09-17 20:49:52.359308528 +0000 UTC m=+7.893248318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnxn" (UniqueName: "kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn") pod "network-check-target-v57r9" (UID: "be6cd316-29b1-4d56-9ae4-f9688eba78ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:50.575044 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.574975    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 20:44:48 +0000 UTC" deadline="2028-04-24 01:27:34.905217516 +0000 UTC"
Sep 17 20:49:50.575044 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.575042    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14020h37m44.330179411s"
Sep 17 20:49:50.668547 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.668513    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:50.668723 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.668659    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:49:50.669073 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.669051    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:50.669173 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:50.669152    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:49:50.725407 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.725337    2586 generic.go:356] "Generic (PLEG): container finished" podID="7be62ecf7e2d5494a2c86866674c5e01" containerID="cd9dedf7e1c0c53fcabdf287b13b54ada1d1a13b893a070e6e74add526d1a526" exitCode=0
Sep 17 20:49:50.725956 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.725900    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal" event={"ID":"7be62ecf7e2d5494a2c86866674c5e01","Type":"ContainerDied","Data":"cd9dedf7e1c0c53fcabdf287b13b54ada1d1a13b893a070e6e74add526d1a526"}
Sep 17 20:49:50.765798 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:50.765539    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-254.ec2.internal" podStartSLOduration=3.765515953 podStartE2EDuration="3.765515953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:49.705695886 +0000 UTC m=+5.239635694" watchObservedRunningTime="2026-09-17 20:49:50.765515953 +0000 UTC m=+6.299455763"
Sep 17 20:49:51.666054 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:51.666022    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:51.666620 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:51.666150    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:49:51.751912 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:51.751856    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal" event={"ID":"7be62ecf7e2d5494a2c86866674c5e01","Type":"ContainerStarted","Data":"9b53faffffbe7edd94ffb4f0ed0ab1c6311176ba5b9761f59a18cb8bf421f3f2"}
Sep 17 20:49:52.174339 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:52.173851    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:52.174339 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.174033    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:52.174339 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.174096    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.174077249 +0000 UTC m=+11.708017061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:52.274644 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:52.274599    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:52.274831 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.274761    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:52.274831 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.274825    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret podName:32ea5724-e1d9-4592-8b7c-32e9d08b5388 nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.27480768 +0000 UTC m=+11.808747473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret") pod "global-pull-secret-syncer-x4x76" (UID: "32ea5724-e1d9-4592-8b7c-32e9d08b5388") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:52.375400 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:52.375316    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:52.375589 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.375489    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:52.375589 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.375513    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:52.375589 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.375526    2586 projected.go:196] Error preparing data for projected volume kube-api-access-nxnxn for pod openshift-network-diagnostics/network-check-target-v57r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:52.375589 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.375587    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn podName:be6cd316-29b1-4d56-9ae4-f9688eba78ec nodeName:}" failed. No retries permitted until 2026-09-17 20:49:56.375569139 +0000 UTC m=+11.909508931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnxn" (UniqueName: "kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn") pod "network-check-target-v57r9" (UID: "be6cd316-29b1-4d56-9ae4-f9688eba78ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:52.667022 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:52.666410    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:52.667022 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.666562    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:49:52.667920 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:52.667628    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:52.667920 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:52.667749    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:49:53.666913 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:53.666367    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:53.666913 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:53.666512    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:49:54.666537 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:54.666503    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:54.666742 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:54.666651    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:49:54.666742 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:54.666664    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:54.666876 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:54.666771    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:49:55.666009 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:55.665975    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:55.666481 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:55.666104    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:49:56.209906 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:56.209870    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:56.210078 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.210000    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:56.210078 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.210069    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:04.210050903 +0000 UTC m=+19.743990704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:49:56.311469 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:56.311426    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:56.311636 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.311608    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:56.311706 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.311674    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret podName:32ea5724-e1d9-4592-8b7c-32e9d08b5388 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:04.311655962 +0000 UTC m=+19.845595752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret") pod "global-pull-secret-syncer-x4x76" (UID: "32ea5724-e1d9-4592-8b7c-32e9d08b5388") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:49:56.412449 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:56.412408    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:56.412663 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.412639    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:49:56.412663 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.412667    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:49:56.412837 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.412682    2586 projected.go:196] Error preparing data for projected volume kube-api-access-nxnxn for pod openshift-network-diagnostics/network-check-target-v57r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:56.412837 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.412748    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn podName:be6cd316-29b1-4d56-9ae4-f9688eba78ec nodeName:}" failed. No retries permitted until 2026-09-17 20:50:04.412729198 +0000 UTC m=+19.946668996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnxn" (UniqueName: "kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn") pod "network-check-target-v57r9" (UID: "be6cd316-29b1-4d56-9ae4-f9688eba78ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:49:56.667279 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:56.667157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:56.667715 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.667280    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:49:56.667715 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:56.667393    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:56.667715 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:56.667526    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:49:57.666772 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:57.666724    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:57.666932 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:57.666870    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:49:58.666775 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:58.666736    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:49:58.667399 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:58.666735    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:49:58.667399 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:58.666885    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:49:58.667399 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:58.666963    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:49:59.666736 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:49:59.666697    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:49:59.666922 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:49:59.666829    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:00.666460 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:00.666416    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:00.666460 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:00.666446    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:00.666699 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:00.666560    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:00.666699 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:00.666620    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:01.666128 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:01.666090    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:01.666575 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:01.666234    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:02.666655 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:02.666620    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:02.667131 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:02.666761    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:02.667131 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:02.666825    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:02.667131 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:02.666949    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:03.666462 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:03.666422    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:03.666667 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:03.666536    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:04.269918 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:04.269878    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:04.270426 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.270004    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:04.270426 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.270059    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:20.270044567 +0000 UTC m=+35.803984354 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:04.370706 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:04.370664    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:04.370894 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.370804    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:04.370894 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.370884    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret podName:32ea5724-e1d9-4592-8b7c-32e9d08b5388 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:20.370864053 +0000 UTC m=+35.904803841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret") pod "global-pull-secret-syncer-x4x76" (UID: "32ea5724-e1d9-4592-8b7c-32e9d08b5388") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:04.471837 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:04.471789    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:04.471990 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.471971    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:50:04.472054 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.471996    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:50:04.472054 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.472010    2586 projected.go:196] Error preparing data for projected volume kube-api-access-nxnxn for pod openshift-network-diagnostics/network-check-target-v57r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:04.472132 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.472078    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn podName:be6cd316-29b1-4d56-9ae4-f9688eba78ec nodeName:}" failed. No retries permitted until 2026-09-17 20:50:20.472059844 +0000 UTC m=+36.005999669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnxn" (UniqueName: "kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn") pod "network-check-target-v57r9" (UID: "be6cd316-29b1-4d56-9ae4-f9688eba78ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:04.666120 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:04.666045    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:04.666296 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.666190    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:04.666296 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:04.666245    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:04.666428 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:04.666370    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:05.666752 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:05.666718    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:05.667193 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:05.666842    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:06.667279 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:06.667244    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:06.667761 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:06.667334    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:06.667761 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:06.667388    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:06.667761 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:06.667473    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:07.666010 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.665801    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:07.666167 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:07.666098    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:07.787753 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.787566    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:50:07.788596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.788116    2586 generic.go:356] "Generic (PLEG): container finished" podID="818325ca-b623-4f67-8664-9e8d89e50ddc" containerID="72058900de390b54660df2d54aecc0bd95f65111868665881735c4e655ff2766" exitCode=1
Sep 17 20:50:07.788596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.788189    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"f37cd391279712899d10b8fc46c41480bb288b27d09b0e0df26453f08c0c51af"}
Sep 17 20:50:07.788596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.788263    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"43b06fe947434755ce6a96c48fb140f59c5f4d5e829ac0cebf24e5fdf6f94b0a"}
Sep 17 20:50:07.788596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.788299    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"91f225de4b3a2d1dbd656ad4fb67bc7419e9183069150f6026c2cf2afb268961"}
Sep 17 20:50:07.788596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.788343    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"60f330fef78313b7a1e29ae3611d7e9f5fc1599990f0851c0ce9e7ca3bbbf4ce"}
Sep 17 20:50:07.788596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.788385    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerDied","Data":"72058900de390b54660df2d54aecc0bd95f65111868665881735c4e655ff2766"}
Sep 17 20:50:07.788596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.788423    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"1866bbff022fba04fc77e6f881da8ce8f525a53f0d9a6ceeed789da692e54cb5"}
Sep 17 20:50:07.789607 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.789586    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wlqs4" event={"ID":"a2c6d687-7ce5-41d2-beea-c8e7833fc4e0","Type":"ContainerStarted","Data":"89bac330b22ce9f849d2d41b651ae6e2be1a446ed9602a7769f7e7a7f13aed30"}
Sep 17 20:50:07.791055 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.791034    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fszzw" event={"ID":"459fe0d3-2504-4964-b4ef-4161f8229f68","Type":"ContainerStarted","Data":"53085a6f386304f01603f6ff63473eb5c6b1c9edca2ea1ae8c39713fd34ac729"}
Sep 17 20:50:07.792404 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.792380    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rrt59" event={"ID":"eac2d386-b829-49d2-ab71-42201ebfcf07","Type":"ContainerStarted","Data":"bc603f17b7cc2bcccd26ad5ce605f3596faac2a61e5a1c678749e21d50cdfcff"}
Sep 17 20:50:07.793903 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.793886    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9" event={"ID":"c4292a64-fe58-4798-b5a2-37dbcbeae4bc","Type":"ContainerStarted","Data":"f3fa30e4510653677fdaf5cf946b65b04806084097dfb40ff0915b3e17a00605"}
Sep 17 20:50:07.795270 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.795250    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7c4kq" event={"ID":"8f3930b3-92c5-4083-94fe-1881c5969abd","Type":"ContainerStarted","Data":"04657f1c76eba2b044bb23646a70f9ea59ca152c334e479d33841848ff7f22a9"}
Sep 17 20:50:07.796711 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.796690    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p8cpg" event={"ID":"5174cf12-96bf-4526-b5ee-66dcc04371c0","Type":"ContainerStarted","Data":"5d98f93642082ed1de8272b71c4e31977e3ed3204a150c8c3b2798c93938a662"}
Sep 17 20:50:07.798161 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.798140    2586 generic.go:356] "Generic (PLEG): container finished" podID="5200bd98-2df0-453a-9ea0-251c997ede89" containerID="785e968b123976e4f5cb43f580348ba347772ee7d796957f19a4acc1ae5cb00a" exitCode=0
Sep 17 20:50:07.798243 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.798168    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerDied","Data":"785e968b123976e4f5cb43f580348ba347772ee7d796957f19a4acc1ae5cb00a"}
Sep 17 20:50:07.804754 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.804716    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-254.ec2.internal" podStartSLOduration=20.804703465 podStartE2EDuration="20.804703465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:49:51.768572338 +0000 UTC m=+7.302512149" watchObservedRunningTime="2026-09-17 20:50:07.804703465 +0000 UTC m=+23.338643273"
Sep 17 20:50:07.805106 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.805070    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wlqs4" podStartSLOduration=4.587364635 podStartE2EDuration="21.805059923s" totalImagesPullingTime="17.217695288s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:07.80452612 +0000 UTC m=+23.338465929" watchObservedRunningTime="2026-09-17 20:50:07.805059923 +0000 UTC m=+23.338999733"
Sep 17 20:50:07.839990 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.839942    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rrt59" podStartSLOduration=8.667290275 podStartE2EDuration="21.839925972s" totalImagesPullingTime="13.172635697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:07.839624953 +0000 UTC m=+23.373564762" watchObservedRunningTime="2026-09-17 20:50:07.839925972 +0000 UTC m=+23.373865781"
Sep 17 20:50:07.856821 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.856767    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fszzw" podStartSLOduration=4.628233588 podStartE2EDuration="21.856745788s" totalImagesPullingTime="17.2285122s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:07.856293899 +0000 UTC m=+23.390233708" watchObservedRunningTime="2026-09-17 20:50:07.856745788 +0000 UTC m=+23.390685597"
Sep 17 20:50:07.877507 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.877452    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7c4kq" podStartSLOduration=4.64711259 podStartE2EDuration="21.877435614s" totalImagesPullingTime="17.230323024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:07.877282075 +0000 UTC m=+23.411221885" watchObservedRunningTime="2026-09-17 20:50:07.877435614 +0000 UTC m=+23.411375422"
Sep 17 20:50:07.895069 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:07.895016    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p8cpg" podStartSLOduration=4.663944802 podStartE2EDuration="21.894997544s" totalImagesPullingTime="17.231052742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:07.894265279 +0000 UTC m=+23.428205088" watchObservedRunningTime="2026-09-17 20:50:07.894997544 +0000 UTC m=+23.428937379"
Sep 17 20:50:08.063176 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.063126    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 20:50:08.610988 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.610860    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T20:50:08.063153569Z","UUID":"6d22e5e4-1f1f-4d57-969c-c780bd0d155d","Handler":null,"Name":"","Endpoint":""}
Sep 17 20:50:08.612549 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.612528    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 20:50:08.612684 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.612563    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 20:50:08.666435 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.666398    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:08.666620 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.666404    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:08.666620 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:08.666551    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:08.666719 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:08.666661    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:08.805166 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.805126    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wjnf2" event={"ID":"078730b6-ff30-4d6b-9146-451243e73b49","Type":"ContainerStarted","Data":"ff1d6d5d72087a2b659835805a6afbc9fd10bc81e8fbc9da1d2454bf5ea0b136"}
Sep 17 20:50:08.808182 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.808086    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9" event={"ID":"c4292a64-fe58-4798-b5a2-37dbcbeae4bc","Type":"ContainerStarted","Data":"8b5c457843d94300ab2399e79a4ecfb9e3fafeff5210d87a785d2352286dcdec"}
Sep 17 20:50:08.826119 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:08.826069    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wjnf2" podStartSLOduration=4.597746661 podStartE2EDuration="21.826051843s" totalImagesPullingTime="17.228305182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:08.825528432 +0000 UTC m=+24.359468241" watchObservedRunningTime="2026-09-17 20:50:08.826051843 +0000 UTC m=+24.359991662"
Sep 17 20:50:09.666682 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:09.666648    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:09.666911 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:09.666760    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:09.814520 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:09.814489    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:50:09.815034 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:09.814946    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"6c0565e470122221f3c03ac1920b107d4a5339dd3a691997d8a3450cb3cce682"}
Sep 17 20:50:09.818059 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:09.818025    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9" event={"ID":"c4292a64-fe58-4798-b5a2-37dbcbeae4bc","Type":"ContainerStarted","Data":"3e27f9a0030b0ac84816bda02313f8af2a7fb08fde01ab28dbb2b91289191551"}
Sep 17 20:50:10.666077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:10.666039    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:10.666077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:10.666065    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:10.666335 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:10.666174    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:10.666335 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:10.666255    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:11.666051 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:11.666015    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:11.666451 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:11.666124    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:11.725872 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:11.725832    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:50:11.726427 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:11.726407    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:50:11.743879 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:11.743814    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r74s9" podStartSLOduration=6.590292271 podStartE2EDuration="25.74379646s" totalImagesPullingTime="19.153504189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:09.843739973 +0000 UTC m=+25.377679783" watchObservedRunningTime="2026-09-17 20:50:11.74379646 +0000 UTC m=+27.277736286"
Sep 17 20:50:11.821730 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:11.821536    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:50:11.822030 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:11.822014    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rrt59"
Sep 17 20:50:12.666779 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:12.666702    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:12.667468 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:12.666880    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:12.667468 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:12.666920    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:12.667468 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:12.666993    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:12.825746 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:12.825714    2586 generic.go:356] "Generic (PLEG): container finished" podID="5200bd98-2df0-453a-9ea0-251c997ede89" containerID="187ada03c3547e94e4e77e8b3968c0ce5929a389174b8c9519805e20ec5ac27e" exitCode=0
Sep 17 20:50:12.825918 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:12.825796    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerDied","Data":"187ada03c3547e94e4e77e8b3968c0ce5929a389174b8c9519805e20ec5ac27e"}
Sep 17 20:50:12.829408 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:12.829388    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:50:12.829754 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:12.829731    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"9b2f269ae7b99025c44a8784ea34e16ea5565598624e092a0c03d3bc7015c445"}
Sep 17 20:50:12.830159 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:12.830146    2586 scope.go:117] "RemoveContainer" containerID="72058900de390b54660df2d54aecc0bd95f65111868665881735c4e655ff2766"
Sep 17 20:50:13.665947 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.665764    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:13.666113 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:13.666022    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:13.835334 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.835308    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:50:13.835722 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.835699    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" event={"ID":"818325ca-b623-4f67-8664-9e8d89e50ddc","Type":"ContainerStarted","Data":"121739960f12512e87a84fc57091e50aca4c9a23fe863c5436b6ed90849d7c03"}
Sep 17 20:50:13.842209 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.841845    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:50:13.842209 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.841920    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:50:13.842209 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.841975    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:50:13.860371 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.860332    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:50:13.860552 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.860537    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:50:13.874071 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:13.874007    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d" podStartSLOduration=9.646340239 podStartE2EDuration="26.873983295s" totalImagesPullingTime="17.227643056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:13.870436818 +0000 UTC m=+29.404376629" watchObservedRunningTime="2026-09-17 20:50:13.873983295 +0000 UTC m=+29.407923107"
Sep 17 20:50:14.017885 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.017852    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4x76"]
Sep 17 20:50:14.018031 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.018021    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:14.018124 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:14.018105    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:14.021255 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.021230    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v57r9"]
Sep 17 20:50:14.021398 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.021330    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:14.021457 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:14.021437    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:14.021868 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.021837    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2rvtb"]
Sep 17 20:50:14.021954 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.021921    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:14.022014 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:14.021998    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:14.840404 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.840350    2586 generic.go:356] "Generic (PLEG): container finished" podID="5200bd98-2df0-453a-9ea0-251c997ede89" containerID="98dcb7dd62781817a10bc51d2c3d9e171dbc2345eeb7069a6bf9afb62fe5b12e" exitCode=0
Sep 17 20:50:14.840404 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:14.840389    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerDied","Data":"98dcb7dd62781817a10bc51d2c3d9e171dbc2345eeb7069a6bf9afb62fe5b12e"}
Sep 17 20:50:15.666070 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:15.666044    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:15.666201 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:15.666044    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:15.666201 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:15.666149    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:15.666275 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:15.666044    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:15.666275 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:15.666232    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:15.666330 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:15.666300    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:16.847980 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:16.847898    2586 generic.go:356] "Generic (PLEG): container finished" podID="5200bd98-2df0-453a-9ea0-251c997ede89" containerID="09d555de8134698b6f64db568683df5294d17d792c902abd39a8b02e2a34a8a3" exitCode=0
Sep 17 20:50:16.847980 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:16.847944    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerDied","Data":"09d555de8134698b6f64db568683df5294d17d792c902abd39a8b02e2a34a8a3"}
Sep 17 20:50:17.666595 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:17.666564    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:17.666768 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:17.666686    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:17.666768 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:17.666701    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:17.666913 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:17.666690    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:17.666913 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:17.666856    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:17.666913 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:17.666899    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:19.666022 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:19.665982    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:19.666501 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:19.666039    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:19.666501 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:19.665992    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:19.666501 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:19.666116    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x4x76" podUID="32ea5724-e1d9-4592-8b7c-32e9d08b5388"
Sep 17 20:50:19.666501 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:19.666197    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v57r9" podUID="be6cd316-29b1-4d56-9ae4-f9688eba78ec"
Sep 17 20:50:19.666501 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:19.666290    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2rvtb" podUID="0efd83dd-81e7-487a-b68e-2b9197c5f655"
Sep 17 20:50:20.278156 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.278120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:20.278346 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.278298    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:20.278431 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.278385    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:52.27834951 +0000 UTC m=+67.812289314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 20:50:20.379373 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.379314    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:20.379555 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.379467    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:20.379555 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.379534    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret podName:32ea5724-e1d9-4592-8b7c-32e9d08b5388 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:52.379514527 +0000 UTC m=+67.913454313 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret") pod "global-pull-secret-syncer-x4x76" (UID: "32ea5724-e1d9-4592-8b7c-32e9d08b5388") : object "kube-system"/"original-pull-secret" not registered
Sep 17 20:50:20.480298 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.480228    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:20.480500 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.480407    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 20:50:20.480500 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.480430    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 20:50:20.480500 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.480441    2586 projected.go:196] Error preparing data for projected volume kube-api-access-nxnxn for pod openshift-network-diagnostics/network-check-target-v57r9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:20.480500 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.480493    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn podName:be6cd316-29b1-4d56-9ae4-f9688eba78ec nodeName:}" failed. No retries permitted until 2026-09-17 20:50:52.480479624 +0000 UTC m=+68.014419427 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxnxn" (UniqueName: "kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn") pod "network-check-target-v57r9" (UID: "be6cd316-29b1-4d56-9ae4-f9688eba78ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 20:50:20.684136 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.684110    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-254.ec2.internal" event="NodeReady"
Sep 17 20:50:20.684562 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.684235    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 20:50:20.734702 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.734616    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f479bc4b-mb5qw"]
Sep 17 20:50:20.746290 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.746263    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.749388 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.748991    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 20:50:20.749388 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.749111    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 20:50:20.749388 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.749275    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 20:50:20.749751 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.749735    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vfk5t\""
Sep 17 20:50:20.749891 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.749871    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zc5rd"]
Sep 17 20:50:20.758503 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.758473    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 20:50:20.760522 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.760501    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f479bc4b-mb5qw"]
Sep 17 20:50:20.760622 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.760540    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8xwln"]
Sep 17 20:50:20.760622 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.760611    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.766885 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.766698    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4498\""
Sep 17 20:50:20.767207 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.767023    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 20:50:20.767207 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.767050    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 20:50:20.772808 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.772789    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zc5rd"]
Sep 17 20:50:20.772897 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.772879    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:20.777401 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.777384    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 20:50:20.777651 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.777631    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 20:50:20.777745 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.777655    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 20:50:20.777745 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.777680    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wc77b\""
Sep 17 20:50:20.787891 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.787870    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8xwln"]
Sep 17 20:50:20.881842 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.881811    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03e58e0-ebc5-4501-b36f-a94873e29386-tmp-dir\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.881842 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.881841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-bound-sa-token\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.882056 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.881862    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.882056 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.881882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmh5j\" (UniqueName: \"kubernetes.io/projected/f03e58e0-ebc5-4501-b36f-a94873e29386-kube-api-access-mmh5j\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.882056 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.881896    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-installation-pull-secrets\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.882056 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.881994    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-certificates\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.882056 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882027    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e58e0-ebc5-4501-b36f-a94873e29386-config-volume\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.882056 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-image-registry-private-configuration\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.882349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882071    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-trusted-ca\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.882349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882123    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d8738bd-241e-49cc-88d3-758ad44cef5d-ca-trust-extracted\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.882349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882163    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvh9z\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-kube-api-access-bvh9z\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.882349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882197    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7kb\" (UniqueName: \"kubernetes.io/projected/08af7dd3-8619-4aae-a156-128707928ac2-kube-api-access-dr7kb\") pod \"ingress-canary-8xwln\" (UID: \"08af7dd3-8619-4aae-a156-128707928ac2\") " pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:20.882349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882238    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08af7dd3-8619-4aae-a156-128707928ac2-cert\") pod \"ingress-canary-8xwln\" (UID: \"08af7dd3-8619-4aae-a156-128707928ac2\") " pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:20.882349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.882265    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.983270 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e58e0-ebc5-4501-b36f-a94873e29386-config-volume\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.983270 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983274    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-image-registry-private-configuration\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.983555 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983299    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-trusted-ca\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.983555 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983416    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d8738bd-241e-49cc-88d3-758ad44cef5d-ca-trust-extracted\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.983555 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983453    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvh9z\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-kube-api-access-bvh9z\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.983713 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983633    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7kb\" (UniqueName: \"kubernetes.io/projected/08af7dd3-8619-4aae-a156-128707928ac2-kube-api-access-dr7kb\") pod \"ingress-canary-8xwln\" (UID: \"08af7dd3-8619-4aae-a156-128707928ac2\") " pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:20.983713 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983686    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08af7dd3-8619-4aae-a156-128707928ac2-cert\") pod \"ingress-canary-8xwln\" (UID: \"08af7dd3-8619-4aae-a156-128707928ac2\") " pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:20.983811 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983715    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.983811 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983742    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03e58e0-ebc5-4501-b36f-a94873e29386-tmp-dir\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.983811 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-bound-sa-token\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.983811 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983793    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983826    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmh5j\" (UniqueName: \"kubernetes.io/projected/f03e58e0-ebc5-4501-b36f-a94873e29386-kube-api-access-mmh5j\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.983833    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.983850    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f479bc4b-mb5qw: secret "image-registry-tls" not found
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983854    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-installation-pull-secrets\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.983906    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls podName:5d8738bd-241e-49cc-88d3-758ad44cef5d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:21.483886871 +0000 UTC m=+37.017826662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls") pod "image-registry-5f479bc4b-mb5qw" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d") : secret "image-registry-tls" not found
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983945    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-certificates\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983792    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e58e0-ebc5-4501-b36f-a94873e29386-config-volume\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.984012 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.984008    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f03e58e0-ebc5-4501-b36f-a94873e29386-tmp-dir\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.984451 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.984098    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:20.984451 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:20.984144    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls podName:f03e58e0-ebc5-4501-b36f-a94873e29386 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:21.48412875 +0000 UTC m=+37.018068559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls") pod "dns-default-zc5rd" (UID: "f03e58e0-ebc5-4501-b36f-a94873e29386") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:20.984451 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.983826    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d8738bd-241e-49cc-88d3-758ad44cef5d-ca-trust-extracted\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.984451 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.984291    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-trusted-ca\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.984451 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.984432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-certificates\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.988536 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.988395    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-installation-pull-secrets\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.988536 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.988407    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-image-registry-private-configuration\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.993742 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.993718    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08af7dd3-8619-4aae-a156-128707928ac2-cert\") pod \"ingress-canary-8xwln\" (UID: \"08af7dd3-8619-4aae-a156-128707928ac2\") " pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:20.994059 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.994036    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7kb\" (UniqueName: \"kubernetes.io/projected/08af7dd3-8619-4aae-a156-128707928ac2-kube-api-access-dr7kb\") pod \"ingress-canary-8xwln\" (UID: \"08af7dd3-8619-4aae-a156-128707928ac2\") " pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:20.994833 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.994813    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmh5j\" (UniqueName: \"kubernetes.io/projected/f03e58e0-ebc5-4501-b36f-a94873e29386-kube-api-access-mmh5j\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:20.995008 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.994990    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-bound-sa-token\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:20.995277 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:20.995261    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvh9z\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-kube-api-access-bvh9z\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:21.081785 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.081752    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8xwln"
Sep 17 20:50:21.471463 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.471429    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8xwln"]
Sep 17 20:50:21.474160 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:50:21.474133    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08af7dd3_8619_4aae_a156_128707928ac2.slice/crio-c74e46fd51af1475ee15dd87ec5b4b520b027330acf44f1f202c93fd4cb090f8 WatchSource:0}: Error finding container c74e46fd51af1475ee15dd87ec5b4b520b027330acf44f1f202c93fd4cb090f8: Status 404 returned error can't find the container with id c74e46fd51af1475ee15dd87ec5b4b520b027330acf44f1f202c93fd4cb090f8
Sep 17 20:50:21.488333 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.488313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:21.488437 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.488420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:21.488504 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:21.488482    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:21.488560 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:21.488541    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:21.488596 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:21.488565    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f479bc4b-mb5qw: secret "image-registry-tls" not found
Sep 17 20:50:21.488596 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:21.488570    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls podName:f03e58e0-ebc5-4501-b36f-a94873e29386 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:22.488548887 +0000 UTC m=+38.022488681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls") pod "dns-default-zc5rd" (UID: "f03e58e0-ebc5-4501-b36f-a94873e29386") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:21.488681 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:21.488629    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls podName:5d8738bd-241e-49cc-88d3-758ad44cef5d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:22.488611006 +0000 UTC m=+38.022550816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls") pod "image-registry-5f479bc4b-mb5qw" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d") : secret "image-registry-tls" not found
Sep 17 20:50:21.666316 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.666281    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:21.666502 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.666326    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:21.666502 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.666295    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:21.668984 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.668966    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:21.669174 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.669032    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:21.669174 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.669074    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:21.669174 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.669113    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kfcvc\""
Sep 17 20:50:21.669318 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.669209    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:21.669439 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.669426    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9vrtf\""
Sep 17 20:50:21.854145 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.854071    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"]
Sep 17 20:50:21.862558 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.862535    2586 generic.go:356] "Generic (PLEG): container finished" podID="5200bd98-2df0-453a-9ea0-251c997ede89" containerID="e2fba47ff31ac3cac6030db05990789cd5bf1fbbf0294dbf61924b7639d21f7a" exitCode=0
Sep 17 20:50:21.868181 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.868158    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerDied","Data":"e2fba47ff31ac3cac6030db05990789cd5bf1fbbf0294dbf61924b7639d21f7a"}
Sep 17 20:50:21.868274 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.868194    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8xwln" event={"ID":"08af7dd3-8619-4aae-a156-128707928ac2","Type":"ContainerStarted","Data":"c74e46fd51af1475ee15dd87ec5b4b520b027330acf44f1f202c93fd4cb090f8"}
Sep 17 20:50:21.868274 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.868212    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"]
Sep 17 20:50:21.868414 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.868308    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:21.870876 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.870844    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-56m8s\""
Sep 17 20:50:21.872966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.872518    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 20:50:21.872966 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.872802    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 20:50:21.873269 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.873249    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 20:50:21.875805 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.875788    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 20:50:21.885396 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.884457    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"]
Sep 17 20:50:21.885396 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.885300    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:21.887914 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.887893    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 20:50:21.903058 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.903039    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"]
Sep 17 20:50:21.903150 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.903066    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"]
Sep 17 20:50:21.903150 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.903088    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"]
Sep 17 20:50:21.903261 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.903159    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:21.907164 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.906887    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 20:50:21.907164 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.907150    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 20:50:21.907164 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.907163    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 20:50:21.907492 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.907256    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 20:50:21.993017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.992821    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqdc\" (UniqueName: \"kubernetes.io/projected/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-kube-api-access-jgqdc\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:21.993017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.992877    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b865072-e048-41b0-90dc-902f75a7077a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq\" (UID: \"7b865072-e048-41b0-90dc-902f75a7077a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:21.993017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.992927    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-hub\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:21.993017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.992955    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87lb\" (UniqueName: \"kubernetes.io/projected/7b865072-e048-41b0-90dc-902f75a7077a-kube-api-access-h87lb\") pod \"managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq\" (UID: \"7b865072-e048-41b0-90dc-902f75a7077a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:21.993017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.992981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-tmp\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:21.993017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.993005    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:21.993441 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.993079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:21.993441 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.993175    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b87m\" (UniqueName: \"kubernetes.io/projected/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-kube-api-access-6b87m\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:21.993441 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.993326    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:21.993666 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.993604    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:21.993666 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:21.993639    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-ca\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.094712 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.094682    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqdc\" (UniqueName: \"kubernetes.io/projected/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-kube-api-access-jgqdc\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:22.094881 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.094728    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b865072-e048-41b0-90dc-902f75a7077a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq\" (UID: \"7b865072-e048-41b0-90dc-902f75a7077a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:22.094881 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.094749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-hub\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.095001 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.094884    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87lb\" (UniqueName: \"kubernetes.io/projected/7b865072-e048-41b0-90dc-902f75a7077a-kube-api-access-h87lb\") pod \"managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq\" (UID: \"7b865072-e048-41b0-90dc-902f75a7077a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:22.095001 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.094927    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-tmp\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:22.095001 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.094955    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.095001 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.094980    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.095195 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.095005    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b87m\" (UniqueName: \"kubernetes.io/projected/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-kube-api-access-6b87m\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.095195 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.095086    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.095195 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.095156    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:22.095195 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.095171    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-ca\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.095410 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.095388    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-tmp\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:22.096538 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.096504    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.098131 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.098089    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-ca\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.098456 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.098342    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7b865072-e048-41b0-90dc-902f75a7077a-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq\" (UID: \"7b865072-e048-41b0-90dc-902f75a7077a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:22.098850 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.098817    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-hub\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.098850 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.098849    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:22.099017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.098942    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.099017 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.098999    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.104302 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.104232    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqdc\" (UniqueName: \"kubernetes.io/projected/43b62285-0ca9-4904-87e6-af6d5d3c9b9d-kube-api-access-jgqdc\") pod \"klusterlet-addon-workmgr-6f5b9bbf7f-txk5g\" (UID: \"43b62285-0ca9-4904-87e6-af6d5d3c9b9d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:22.104419 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.104386    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b87m\" (UniqueName: \"kubernetes.io/projected/f5e84afe-be9d-4e18-96eb-5ce91cfebf15-kube-api-access-6b87m\") pod \"cluster-proxy-proxy-agent-79b55f68ff-898tk\" (UID: \"f5e84afe-be9d-4e18-96eb-5ce91cfebf15\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.104720 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.104697    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87lb\" (UniqueName: \"kubernetes.io/projected/7b865072-e048-41b0-90dc-902f75a7077a-kube-api-access-h87lb\") pod \"managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq\" (UID: \"7b865072-e048-41b0-90dc-902f75a7077a\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:22.186617 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.186579    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"
Sep 17 20:50:22.196798 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.196767    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:22.213960 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.213919    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:50:22.489229 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.489011    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq"]
Sep 17 20:50:22.492112 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:50:22.492082    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b865072_e048_41b0_90dc_902f75a7077a.slice/crio-7f673c3c67d62dbf3e5d2f17e7fa6b95d139c61a56bc4567c81f41b60dc3b760 WatchSource:0}: Error finding container 7f673c3c67d62dbf3e5d2f17e7fa6b95d139c61a56bc4567c81f41b60dc3b760: Status 404 returned error can't find the container with id 7f673c3c67d62dbf3e5d2f17e7fa6b95d139c61a56bc4567c81f41b60dc3b760
Sep 17 20:50:22.499137 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.499111    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:22.499220 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.499164    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:22.499274 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:22.499256    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:22.499319 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:22.499276    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f479bc4b-mb5qw: secret "image-registry-tls" not found
Sep 17 20:50:22.499319 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:22.499297    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:22.499495 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:22.499346    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls podName:5d8738bd-241e-49cc-88d3-758ad44cef5d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:24.499325351 +0000 UTC m=+40.033265152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls") pod "image-registry-5f479bc4b-mb5qw" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d") : secret "image-registry-tls" not found
Sep 17 20:50:22.499495 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:22.499378    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls podName:f03e58e0-ebc5-4501-b36f-a94873e29386 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:24.499367251 +0000 UTC m=+40.033307042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls") pod "dns-default-zc5rd" (UID: "f03e58e0-ebc5-4501-b36f-a94873e29386") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:22.513120 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.513094    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"]
Sep 17 20:50:22.516025 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:50:22.515988    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b62285_0ca9_4904_87e6_af6d5d3c9b9d.slice/crio-a93bbba59d92bfaf4e2c113d01b26c41d585e93c75a4a6a56cdf6a9052c1a5ce WatchSource:0}: Error finding container a93bbba59d92bfaf4e2c113d01b26c41d585e93c75a4a6a56cdf6a9052c1a5ce: Status 404 returned error can't find the container with id a93bbba59d92bfaf4e2c113d01b26c41d585e93c75a4a6a56cdf6a9052c1a5ce
Sep 17 20:50:22.553767 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.553728    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"]
Sep 17 20:50:22.556409 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:50:22.556378    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5e84afe_be9d_4e18_96eb_5ce91cfebf15.slice/crio-79eed3cb38c7642787cd9a1eb9b0077c3eb80102074815b336c02c1591a1a7ca WatchSource:0}: Error finding container 79eed3cb38c7642787cd9a1eb9b0077c3eb80102074815b336c02c1591a1a7ca: Status 404 returned error can't find the container with id 79eed3cb38c7642787cd9a1eb9b0077c3eb80102074815b336c02c1591a1a7ca
Sep 17 20:50:22.869483 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.869455    2586 generic.go:356] "Generic (PLEG): container finished" podID="5200bd98-2df0-453a-9ea0-251c997ede89" containerID="ceccb12409d9e6f6fe5ae80048ba1df6d24ceb61a6d44053d2da8040f504d4cc" exitCode=0
Sep 17 20:50:22.869996 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.869530    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerDied","Data":"ceccb12409d9e6f6fe5ae80048ba1df6d24ceb61a6d44053d2da8040f504d4cc"}
Sep 17 20:50:22.870764 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.870743    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g" event={"ID":"43b62285-0ca9-4904-87e6-af6d5d3c9b9d","Type":"ContainerStarted","Data":"a93bbba59d92bfaf4e2c113d01b26c41d585e93c75a4a6a56cdf6a9052c1a5ce"}
Sep 17 20:50:22.871944 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.871914    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" event={"ID":"f5e84afe-be9d-4e18-96eb-5ce91cfebf15","Type":"ContainerStarted","Data":"79eed3cb38c7642787cd9a1eb9b0077c3eb80102074815b336c02c1591a1a7ca"}
Sep 17 20:50:22.872940 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:22.872922    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq" event={"ID":"7b865072-e048-41b0-90dc-902f75a7077a","Type":"ContainerStarted","Data":"7f673c3c67d62dbf3e5d2f17e7fa6b95d139c61a56bc4567c81f41b60dc3b760"}
Sep 17 20:50:23.878898 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:23.878863    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d85tw" event={"ID":"5200bd98-2df0-453a-9ea0-251c997ede89","Type":"ContainerStarted","Data":"9712c845c63f79e7fb193a61e07dc9e2bd82574ea32aa811ebe31b7145f42f2c"}
Sep 17 20:50:23.906046 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:23.905972    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d85tw" podStartSLOduration=10.032834945 podStartE2EDuration="37.905957022s" totalImagesPullingTime="27.873122077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:23.905548891 +0000 UTC m=+39.439488713" watchObservedRunningTime="2026-09-17 20:50:23.905957022 +0000 UTC m=+39.439896830"
Sep 17 20:50:24.519516 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:24.519472    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:24.519706 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:24.519599    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:24.519706 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:24.519611    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:24.519706 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:24.519675    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls podName:f03e58e0-ebc5-4501-b36f-a94873e29386 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:28.51965316 +0000 UTC m=+44.053592954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls") pod "dns-default-zc5rd" (UID: "f03e58e0-ebc5-4501-b36f-a94873e29386") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:24.519906 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:24.519722    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:24.519906 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:24.519736    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f479bc4b-mb5qw: secret "image-registry-tls" not found
Sep 17 20:50:24.519906 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:24.519794    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls podName:5d8738bd-241e-49cc-88d3-758ad44cef5d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:28.519778809 +0000 UTC m=+44.053718599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls") pod "image-registry-5f479bc4b-mb5qw" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d") : secret "image-registry-tls" not found
Sep 17 20:50:25.889718 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:25.889317    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq" event={"ID":"7b865072-e048-41b0-90dc-902f75a7077a","Type":"ContainerStarted","Data":"be4191b3678f2e2fb80f6803a06f88875d9f02d157bb6a7bf88c91f8baefb3c9"}
Sep 17 20:50:25.891830 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:25.891803    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8xwln" event={"ID":"08af7dd3-8619-4aae-a156-128707928ac2","Type":"ContainerStarted","Data":"0e77e82e58c35ecfa8d8c22e5aef7ef29dd51ec2657f52555d022c4376a9ddd7"}
Sep 17 20:50:25.905704 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:25.905656    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ff6d9fdc4-swwsq" podStartSLOduration=2.226729368 podStartE2EDuration="4.905638119s" totalImagesPullingTime="2.678908751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:25.904559443 +0000 UTC m=+41.438499253" watchObservedRunningTime="2026-09-17 20:50:25.905638119 +0000 UTC m=+41.439577929"
Sep 17 20:50:25.921475 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:25.921410    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8xwln" podStartSLOduration=2.278596956 podStartE2EDuration="5.921391453s" totalImagesPullingTime="3.642794497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:25.919997294 +0000 UTC m=+41.453937104" watchObservedRunningTime="2026-09-17 20:50:25.921391453 +0000 UTC m=+41.455331263"
Sep 17 20:50:28.558616 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:28.558578    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:28.559078 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:28.558629    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:28.559078 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:28.558735    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:28.559078 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:28.558750    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f479bc4b-mb5qw: secret "image-registry-tls" not found
Sep 17 20:50:28.559078 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:28.558813    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls podName:5d8738bd-241e-49cc-88d3-758ad44cef5d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:36.558794449 +0000 UTC m=+52.092734243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls") pod "image-registry-5f479bc4b-mb5qw" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d") : secret "image-registry-tls" not found
Sep 17 20:50:28.559078 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:28.558821    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:28.559078 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:28.558894    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls podName:f03e58e0-ebc5-4501-b36f-a94873e29386 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:36.558875294 +0000 UTC m=+52.092815081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls") pod "dns-default-zc5rd" (UID: "f03e58e0-ebc5-4501-b36f-a94873e29386") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:29.905411 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.905376    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" event={"ID":"f5e84afe-be9d-4e18-96eb-5ce91cfebf15","Type":"ContainerStarted","Data":"13cdd8fe84222edc2ab8adc105e5765a7c1c249bd593a26ad383c936928964fb"}
Sep 17 20:50:29.905782 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.905427    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" event={"ID":"f5e84afe-be9d-4e18-96eb-5ce91cfebf15","Type":"ContainerStarted","Data":"688ac019585ee4a87621b048b1056566b671be6bf11701111bdae2a275d94270"}
Sep 17 20:50:29.905782 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.905462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" event={"ID":"f5e84afe-be9d-4e18-96eb-5ce91cfebf15","Type":"ContainerStarted","Data":"81e606e38d430e2ccaf158e84eebf7dd8f8c610be1fe2debce0552b9de261a24"}
Sep 17 20:50:29.906784 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.906754    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g" event={"ID":"43b62285-0ca9-4904-87e6-af6d5d3c9b9d","Type":"ContainerStarted","Data":"f823e5aa916a1b7527641ff51257cf85ca2411fc17ddc6d31e1dc6890d73fdfa"}
Sep 17 20:50:29.906976 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.906959    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:29.908293 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.908272    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g"
Sep 17 20:50:29.925460 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.925420    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" podStartSLOduration=1.862734338 podStartE2EDuration="8.92540554s" totalImagesPullingTime="7.062671202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:29.924191416 +0000 UTC m=+45.458131225" watchObservedRunningTime="2026-09-17 20:50:29.92540554 +0000 UTC m=+45.459345351"
Sep 17 20:50:29.941037 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:29.941000    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f5b9bbf7f-txk5g" podStartSLOduration=1.847902063 podStartE2EDuration="8.940985715s" totalImagesPullingTime="7.093083652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:29.94014146 +0000 UTC m=+45.474081270" watchObservedRunningTime="2026-09-17 20:50:29.940985715 +0000 UTC m=+45.474925523"
Sep 17 20:50:36.612947 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:36.612904    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:36.612947 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:36.612947    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:36.613397 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:36.613046    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:36.613397 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:36.613063    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:36.613397 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:36.613081    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f479bc4b-mb5qw: secret "image-registry-tls" not found
Sep 17 20:50:36.613397 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:36.613095    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls podName:f03e58e0-ebc5-4501-b36f-a94873e29386 nodeName:}" failed. No retries permitted until 2026-09-17 20:50:52.613080793 +0000 UTC m=+68.147020593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls") pod "dns-default-zc5rd" (UID: "f03e58e0-ebc5-4501-b36f-a94873e29386") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:36.613397 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:36.613129    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls podName:5d8738bd-241e-49cc-88d3-758ad44cef5d nodeName:}" failed. No retries permitted until 2026-09-17 20:50:52.613113253 +0000 UTC m=+68.147053057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls") pod "image-registry-5f479bc4b-mb5qw" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d") : secret "image-registry-tls" not found
Sep 17 20:50:45.857608 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:45.857573    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hg45d"
Sep 17 20:50:52.327748 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.327712    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:50:52.330583 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.330564    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 20:50:52.338266 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:52.338251    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 20:50:52.338323 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:52.338314    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs podName:0efd83dd-81e7-487a-b68e-2b9197c5f655 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:56.338297552 +0000 UTC m=+131.872237339 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs") pod "network-metrics-daemon-2rvtb" (UID: "0efd83dd-81e7-487a-b68e-2b9197c5f655") : secret "metrics-daemon-secret" not found
Sep 17 20:50:52.428906 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.428871    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:52.431427 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.431410    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 20:50:52.442638 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.442614    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/32ea5724-e1d9-4592-8b7c-32e9d08b5388-original-pull-secret\") pod \"global-pull-secret-syncer-x4x76\" (UID: \"32ea5724-e1d9-4592-8b7c-32e9d08b5388\") " pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:52.529193 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.529158    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:52.532131 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.532116    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 20:50:52.543004 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.542988    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 20:50:52.553433 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.553409    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnxn\" (UniqueName: \"kubernetes.io/projected/be6cd316-29b1-4d56-9ae4-f9688eba78ec-kube-api-access-nxnxn\") pod \"network-check-target-v57r9\" (UID: \"be6cd316-29b1-4d56-9ae4-f9688eba78ec\") " pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:52.578583 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.578538    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kfcvc\""
Sep 17 20:50:52.581696 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.581682    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x4x76"
Sep 17 20:50:52.586898 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.586884    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:52.630105 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.630074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:50:52.630254 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.630129    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:50:52.630254 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:52.630246    2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 20:50:52.630369 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:52.630256    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 20:50:52.630369 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:52.630266    2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f479bc4b-mb5qw: secret "image-registry-tls" not found
Sep 17 20:50:52.630369 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:52.630319    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls podName:5d8738bd-241e-49cc-88d3-758ad44cef5d nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.630302471 +0000 UTC m=+100.164242269 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls") pod "image-registry-5f479bc4b-mb5qw" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d") : secret "image-registry-tls" not found
Sep 17 20:50:52.630369 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:50:52.630334    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls podName:f03e58e0-ebc5-4501-b36f-a94873e29386 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:24.630328094 +0000 UTC m=+100.164267881 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls") pod "dns-default-zc5rd" (UID: "f03e58e0-ebc5-4501-b36f-a94873e29386") : secret "dns-default-metrics-tls" not found
Sep 17 20:50:52.859609 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.859587    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x4x76"]
Sep 17 20:50:52.861885 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:50:52.861853    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ea5724_e1d9_4592_8b7c_32e9d08b5388.slice/crio-80456f19cdb050c51ca0a9fc2c872dbd3b8707c4447357412a789659943e44ea WatchSource:0}: Error finding container 80456f19cdb050c51ca0a9fc2c872dbd3b8707c4447357412a789659943e44ea: Status 404 returned error can't find the container with id 80456f19cdb050c51ca0a9fc2c872dbd3b8707c4447357412a789659943e44ea
Sep 17 20:50:52.864349 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.864327    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v57r9"]
Sep 17 20:50:52.867248 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:50:52.867227    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6cd316_29b1_4d56_9ae4_f9688eba78ec.slice/crio-c71868d244206074d6034b9b750adf407f5f1a375543f1ab2f7ecf93c04b64e8 WatchSource:0}: Error finding container c71868d244206074d6034b9b750adf407f5f1a375543f1ab2f7ecf93c04b64e8: Status 404 returned error can't find the container with id c71868d244206074d6034b9b750adf407f5f1a375543f1ab2f7ecf93c04b64e8
Sep 17 20:50:52.976464 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.976432    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v57r9" event={"ID":"be6cd316-29b1-4d56-9ae4-f9688eba78ec","Type":"ContainerStarted","Data":"c71868d244206074d6034b9b750adf407f5f1a375543f1ab2f7ecf93c04b64e8"}
Sep 17 20:50:52.977435 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:52.977409    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4x76" event={"ID":"32ea5724-e1d9-4592-8b7c-32e9d08b5388","Type":"ContainerStarted","Data":"80456f19cdb050c51ca0a9fc2c872dbd3b8707c4447357412a789659943e44ea"}
Sep 17 20:50:58.000315 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:58.000274    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v57r9" event={"ID":"be6cd316-29b1-4d56-9ae4-f9688eba78ec","Type":"ContainerStarted","Data":"a87bb83a14d9bc6f3ac79b6713f862fba15341f7d6369f1ea1b913dbf6e91f74"}
Sep 17 20:50:58.000769 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:58.000383    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:50:58.001776 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:58.001753    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x4x76" event={"ID":"32ea5724-e1d9-4592-8b7c-32e9d08b5388","Type":"ContainerStarted","Data":"d567ece2f975ba468e21f394da24bf8664b34a2c2fa18974e782970b2032c31d"}
Sep 17 20:50:58.017071 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:58.017032    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v57r9" podStartSLOduration=66.764504989 podStartE2EDuration="1m11.01701914s" totalImagesPullingTime="4.252514151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:58.016969484 +0000 UTC m=+73.550909290" watchObservedRunningTime="2026-09-17 20:50:58.01701914 +0000 UTC m=+73.550958948"
Sep 17 20:50:58.032195 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:50:58.032155    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x4x76" podStartSLOduration=66.772681458 podStartE2EDuration="1m11.032142149s" totalImagesPullingTime="4.259460691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:50:58.031527511 +0000 UTC m=+73.565467322" watchObservedRunningTime="2026-09-17 20:50:58.032142149 +0000 UTC m=+73.566081958"
Sep 17 20:51:06.053194 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.053164    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zpxgv"]
Sep 17 20:51:06.056383 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.056341    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.060069 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.060051    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 20:51:06.060786 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.060770    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 20:51:06.060865 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.060772    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qm4s6\""
Sep 17 20:51:06.061030 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.061015    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 20:51:06.061172 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.061049    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 20:51:06.070827 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.070803    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zpxgv"]
Sep 17 20:51:06.131798 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.131767    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2757f4d0-ed48-4bca-81ab-d509f1a74567-crio-socket\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.131798 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.131796    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.131965 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.131816    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.131965 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.131868    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjrx\" (UniqueName: \"kubernetes.io/projected/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-api-access-cpjrx\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.131965 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.131954    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2757f4d0-ed48-4bca-81ab-d509f1a74567-data-volume\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.233130 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.233104    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2757f4d0-ed48-4bca-81ab-d509f1a74567-data-volume\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.233244 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.233162    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2757f4d0-ed48-4bca-81ab-d509f1a74567-crio-socket\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.233244 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.233179    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.233244 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.233199    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.233244 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.233225    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjrx\" (UniqueName: \"kubernetes.io/projected/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-api-access-cpjrx\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.233415 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.233306    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2757f4d0-ed48-4bca-81ab-d509f1a74567-crio-socket\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.233415 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.233310    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.233479 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.233433    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.733412002 +0000 UTC m=+82.267351794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.233479 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.233320    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:06.233554 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.233496    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:06.733484588 +0000 UTC m=+82.267424418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:06.233554 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.233495    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2757f4d0-ed48-4bca-81ab-d509f1a74567-data-volume\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.241096 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.241076    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjrx\" (UniqueName: \"kubernetes.io/projected/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-api-access-cpjrx\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.736150 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.736114    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.736150 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:06.736153    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:06.736349 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.736262    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.736349 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.736324    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:07.736311025 +0000 UTC m=+83.270250816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:06.736450 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.736262    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:06.736450 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:06.736414    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:07.73640092 +0000 UTC m=+83.270340723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:07.742402 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:07.742347    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:07.742402 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:07.742403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:07.742798 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:07.742531    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:07.742798 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:07.742594    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.742579218 +0000 UTC m=+85.276519014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:07.742798 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:07.742599    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:07.742798 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:07.742651    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:09.742639019 +0000 UTC m=+85.276578806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:08.206306 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:08.206284    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wlqs4_a2c6d687-7ce5-41d2-beea-c8e7833fc4e0/dns-node-resolver/0.log"
Sep 17 20:51:08.808050 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:08.808024    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7c4kq_8f3930b3-92c5-4083-94fe-1881c5969abd/node-ca/0.log"
Sep 17 20:51:09.757590 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:09.757557    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:09.757736 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:09.757597    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:09.757736 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:09.757705    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.757804 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:09.757765    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.757751984 +0000 UTC m=+89.291691775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:09.757804 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:09.757704    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.757876 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:09.757842    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:13.757829606 +0000 UTC m=+89.291769409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:09.807489 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:09.807459    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8xwln_08af7dd3-8619-4aae-a156-128707928ac2/serve-healthcheck-canary/0.log"
Sep 17 20:51:13.785337 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:13.785307    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:13.785337 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:13.785340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:13.785756 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:13.785438    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 20:51:13.785756 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:13.785490    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.785477931 +0000 UTC m=+97.319417718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : configmap "kube-rbac-proxy" not found
Sep 17 20:51:13.785756 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:13.785439    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:13.785756 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:13.785577    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:21.785562916 +0000 UTC m=+97.319502709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:21.846801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:21.846756    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:21.846801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:21.846805    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:21.847238 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:21.846894    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:21.847238 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:21.846965    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls podName:2757f4d0-ed48-4bca-81ab-d509f1a74567 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:37.846949055 +0000 UTC m=+113.380888842 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zpxgv" (UID: "2757f4d0-ed48-4bca-81ab-d509f1a74567") : secret "insights-runtime-extractor-tls" not found
Sep 17 20:51:21.847238 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:21.847229    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2757f4d0-ed48-4bca-81ab-d509f1a74567-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:24.666672 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.666633    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:51:24.666672 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.666665    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:51:24.668975 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.668953    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f03e58e0-ebc5-4501-b36f-a94873e29386-metrics-tls\") pod \"dns-default-zc5rd\" (UID: \"f03e58e0-ebc5-4501-b36f-a94873e29386\") " pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:51:24.669209 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.669188    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"image-registry-5f479bc4b-mb5qw\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") " pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:51:24.676490 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.676472    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4498\""
Sep 17 20:51:24.683898 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.683880    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:51:24.960020 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.959954    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zc5rd"]
Sep 17 20:51:24.960520 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.960503    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vfk5t\""
Sep 17 20:51:24.963263 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:51:24.963239    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03e58e0_ebc5_4501_b36f_a94873e29386.slice/crio-fa6c440838d9887e45f775df52749b3a092570e7190ba4ae8d3b9b0020621e23 WatchSource:0}: Error finding container fa6c440838d9887e45f775df52749b3a092570e7190ba4ae8d3b9b0020621e23: Status 404 returned error can't find the container with id fa6c440838d9887e45f775df52749b3a092570e7190ba4ae8d3b9b0020621e23
Sep 17 20:51:24.968997 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:24.968976    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:51:25.082166 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:25.082135    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zc5rd" event={"ID":"f03e58e0-ebc5-4501-b36f-a94873e29386","Type":"ContainerStarted","Data":"fa6c440838d9887e45f775df52749b3a092570e7190ba4ae8d3b9b0020621e23"}
Sep 17 20:51:25.242248 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:25.242172    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f479bc4b-mb5qw"]
Sep 17 20:51:25.245665 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:51:25.245632    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8738bd_241e_49cc_88d3_758ad44cef5d.slice/crio-5d70dd160173aaf520ff1b0e36cad9f5e8511084fc688351e0c76027c7d12fa6 WatchSource:0}: Error finding container 5d70dd160173aaf520ff1b0e36cad9f5e8511084fc688351e0c76027c7d12fa6: Status 404 returned error can't find the container with id 5d70dd160173aaf520ff1b0e36cad9f5e8511084fc688351e0c76027c7d12fa6
Sep 17 20:51:26.086707 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:26.086669    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw" event={"ID":"5d8738bd-241e-49cc-88d3-758ad44cef5d","Type":"ContainerStarted","Data":"c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128"}
Sep 17 20:51:26.087101 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:26.086726    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw" event={"ID":"5d8738bd-241e-49cc-88d3-758ad44cef5d","Type":"ContainerStarted","Data":"5d70dd160173aaf520ff1b0e36cad9f5e8511084fc688351e0c76027c7d12fa6"}
Sep 17 20:51:26.087101 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:26.086819    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:51:26.140390 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:26.140321    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw" podStartSLOduration=99.140305207 podStartE2EDuration="1m39.140305207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:26.139124697 +0000 UTC m=+101.673064528" watchObservedRunningTime="2026-09-17 20:51:26.140305207 +0000 UTC m=+101.674245007"
Sep 17 20:51:27.093597 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:27.093508    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zc5rd" event={"ID":"f03e58e0-ebc5-4501-b36f-a94873e29386","Type":"ContainerStarted","Data":"2f91a6d26449e49948ab30da71d9baed224ba40107972003e42b6b4928b02fca"}
Sep 17 20:51:27.094011 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:27.093597    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:51:27.094011 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:27.093635    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zc5rd" event={"ID":"f03e58e0-ebc5-4501-b36f-a94873e29386","Type":"ContainerStarted","Data":"2fb7c5ea711214f97cbef6701250eb1cce7cc7103ef53f452c9350615cd456b5"}
Sep 17 20:51:29.006809 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:29.006778    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v57r9"
Sep 17 20:51:29.040387 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:29.040319    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zc5rd" podStartSLOduration=67.391957334 podStartE2EDuration="1m9.040298081s" totalImagesPullingTime="1.648340747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:27.130561735 +0000 UTC m=+102.664501544" watchObservedRunningTime="2026-09-17 20:51:29.040298081 +0000 UTC m=+104.574237892"
Sep 17 20:51:37.098760 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:37.098731    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zc5rd"
Sep 17 20:51:37.861166 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:37.861128    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:37.863321 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:37.863295    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2757f4d0-ed48-4bca-81ab-d509f1a74567-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zpxgv\" (UID: \"2757f4d0-ed48-4bca-81ab-d509f1a74567\") " pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:37.865279 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:37.865262    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zpxgv"
Sep 17 20:51:38.132654 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:38.132634    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zpxgv"]
Sep 17 20:51:38.134917 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:51:38.134895    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2757f4d0_ed48_4bca_81ab_d509f1a74567.slice/crio-9402507069df3c6fe1b40f9c565c02b3f78c5492ecf74854646b8b36b1a945a1 WatchSource:0}: Error finding container 9402507069df3c6fe1b40f9c565c02b3f78c5492ecf74854646b8b36b1a945a1: Status 404 returned error can't find the container with id 9402507069df3c6fe1b40f9c565c02b3f78c5492ecf74854646b8b36b1a945a1
Sep 17 20:51:39.131898 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:39.131869    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpxgv" event={"ID":"2757f4d0-ed48-4bca-81ab-d509f1a74567","Type":"ContainerStarted","Data":"383bf776fdd06fc8badc7d388e03703ac34dc436af0af4f7b9a887de69a2f8d1"}
Sep 17 20:51:39.132037 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:39.131918    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpxgv" event={"ID":"2757f4d0-ed48-4bca-81ab-d509f1a74567","Type":"ContainerStarted","Data":"765532a42697444348c1665ac1b6fd8cc50b826657ab05bf6b6f0b8d10904c26"}
Sep 17 20:51:39.132037 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:39.131936    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpxgv" event={"ID":"2757f4d0-ed48-4bca-81ab-d509f1a74567","Type":"ContainerStarted","Data":"9402507069df3c6fe1b40f9c565c02b3f78c5492ecf74854646b8b36b1a945a1"}
Sep 17 20:51:41.139029 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:41.138997    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zpxgv" event={"ID":"2757f4d0-ed48-4bca-81ab-d509f1a74567","Type":"ContainerStarted","Data":"012b61db0f280f8835868149cabede88c5ac8e9fe3bfdbb64b2c1a432a74bbb8"}
Sep 17 20:51:41.159249 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:41.159199    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zpxgv" podStartSLOduration=33.28941731 podStartE2EDuration="35.159183444s" totalImagesPullingTime="1.869766134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:41.157609395 +0000 UTC m=+116.691549205" watchObservedRunningTime="2026-09-17 20:51:41.159183444 +0000 UTC m=+116.693123252"
Sep 17 20:51:42.215873 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:42.215824    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" podUID="f5e84afe-be9d-4e18-96eb-5ce91cfebf15" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 20:51:44.973595 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:44.973560    2586 patch_prober.go:28] interesting pod/image-registry-5f479bc4b-mb5qw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 20:51:44.973976 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:44.973613    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw" podUID="5d8738bd-241e-49cc-88d3-758ad44cef5d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 20:51:47.098776 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:47.098727    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:51:51.569048 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.569018    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f479bc4b-mb5qw"]
Sep 17 20:51:51.848720 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.848638    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pcd5w"]
Sep 17 20:51:51.850693 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.850675    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.853532 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.853506    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 20:51:51.853646 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.853509    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 20:51:51.853646 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.853543    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 20:51:51.853646 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.853570    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 20:51:51.853646 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.853571    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pgb5d\""
Sep 17 20:51:51.854323 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.854307    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 20:51:51.854377 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.854346    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 20:51:51.966486 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966459    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9081b054-8a9e-4345-af29-f353483d47f7-metrics-client-ca\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966486 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966489    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-textfile\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966686 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966506    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-tls\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966686 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966525    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-sys\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966686 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966541    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966686 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966586    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-root\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966833 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966693    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-954kd\" (UniqueName: \"kubernetes.io/projected/9081b054-8a9e-4345-af29-f353483d47f7-kube-api-access-954kd\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966833 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966730    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-wtmp\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:51.966833 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:51.966779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067088 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067055    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-tls\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067227 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067100    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-sys\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067227 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067227 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067134    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-root\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067227 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:52.067154    2586 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 20:51:52.067227 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067179    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-954kd\" (UniqueName: \"kubernetes.io/projected/9081b054-8a9e-4345-af29-f353483d47f7-kube-api-access-954kd\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067227 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067206    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-sys\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067227 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:51:52.067213    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-tls podName:9081b054-8a9e-4345-af29-f353483d47f7 nodeName:}" failed. No retries permitted until 2026-09-17 20:51:52.567192289 +0000 UTC m=+128.101132087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-tls") pod "node-exporter-pcd5w" (UID: "9081b054-8a9e-4345-af29-f353483d47f7") : secret "node-exporter-tls" not found
Sep 17 20:51:52.067597 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067233    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-root\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067597 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067264    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-wtmp\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067597 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067309    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067597 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067334    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9081b054-8a9e-4345-af29-f353483d47f7-metrics-client-ca\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067597 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067386    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-textfile\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067597 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067456    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-wtmp\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067801 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067741    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-textfile\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067906 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067889    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.067945 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.067907    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9081b054-8a9e-4345-af29-f353483d47f7-metrics-client-ca\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.069480 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.069461    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.086101 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.086080    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-954kd\" (UniqueName: \"kubernetes.io/projected/9081b054-8a9e-4345-af29-f353483d47f7-kube-api-access-954kd\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.215696 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.215667    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" podUID="f5e84afe-be9d-4e18-96eb-5ce91cfebf15" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 20:51:52.570716 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.570639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-tls\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.572821 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.572803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9081b054-8a9e-4345-af29-f353483d47f7-node-exporter-tls\") pod \"node-exporter-pcd5w\" (UID: \"9081b054-8a9e-4345-af29-f353483d47f7\") " pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.759424 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:52.759396    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pcd5w"
Sep 17 20:51:52.769191 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:51:52.769160    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9081b054_8a9e_4345_af29_f353483d47f7.slice/crio-0950163bc71d5f3c10bed1136f7c9e778bf42f4513f421fb1efaa4159d05e089 WatchSource:0}: Error finding container 0950163bc71d5f3c10bed1136f7c9e778bf42f4513f421fb1efaa4159d05e089: Status 404 returned error can't find the container with id 0950163bc71d5f3c10bed1136f7c9e778bf42f4513f421fb1efaa4159d05e089
Sep 17 20:51:53.182742 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:53.182701    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcd5w" event={"ID":"9081b054-8a9e-4345-af29-f353483d47f7","Type":"ContainerStarted","Data":"0950163bc71d5f3c10bed1136f7c9e778bf42f4513f421fb1efaa4159d05e089"}
Sep 17 20:51:54.187844 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:54.187809    2586 generic.go:356] "Generic (PLEG): container finished" podID="9081b054-8a9e-4345-af29-f353483d47f7" containerID="8edb56c8dd469b8e8cf1adfdaf7e611e09b5818609beadf3a2a3767d9fddb24d" exitCode=0
Sep 17 20:51:54.188217 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:54.187902    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcd5w" event={"ID":"9081b054-8a9e-4345-af29-f353483d47f7","Type":"ContainerDied","Data":"8edb56c8dd469b8e8cf1adfdaf7e611e09b5818609beadf3a2a3767d9fddb24d"}
Sep 17 20:51:55.193000 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:55.192941    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcd5w" event={"ID":"9081b054-8a9e-4345-af29-f353483d47f7","Type":"ContainerStarted","Data":"8b279f8a740422777a66cba8bb086d2b574517585fdb2a354e6d2d61ff266a65"}
Sep 17 20:51:55.193000 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:55.192992    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pcd5w" event={"ID":"9081b054-8a9e-4345-af29-f353483d47f7","Type":"ContainerStarted","Data":"fdb0f8b61f5d43ab248de594e19b47667e9172f7681e0c83e010eadead2d1ed2"}
Sep 17 20:51:55.212643 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:55.212593    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pcd5w" podStartSLOduration=3.360781505 podStartE2EDuration="4.212578296s" totalImagesPullingTime="791.863582ms" totalInitContainerRuntime="59.933209ms" isStatefulPod=true podCreationTimestamp="2026-09-17 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:55.211146858 +0000 UTC m=+130.745086667" watchObservedRunningTime="2026-09-17 20:51:55.212578296 +0000 UTC m=+130.746518115"
Sep 17 20:51:56.397142 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:56.397096    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:51:56.399284 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:56.399264    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0efd83dd-81e7-487a-b68e-2b9197c5f655-metrics-certs\") pod \"network-metrics-daemon-2rvtb\" (UID: \"0efd83dd-81e7-487a-b68e-2b9197c5f655\") " pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:51:56.488165 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:56.488140    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9vrtf\""
Sep 17 20:51:56.496162 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:56.496146    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2rvtb"
Sep 17 20:51:56.766341 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:56.766315    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2rvtb"]
Sep 17 20:51:56.768161 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:51:56.768136    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efd83dd_81e7_487a_b68e_2b9197c5f655.slice/crio-30a209159416aab33ba5169d211f6ed4de1f757a60ffe3e550c0c08e6cd93fb4 WatchSource:0}: Error finding container 30a209159416aab33ba5169d211f6ed4de1f757a60ffe3e550c0c08e6cd93fb4: Status 404 returned error can't find the container with id 30a209159416aab33ba5169d211f6ed4de1f757a60ffe3e550c0c08e6cd93fb4
Sep 17 20:51:57.200329 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:57.200295    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2rvtb" event={"ID":"0efd83dd-81e7-487a-b68e-2b9197c5f655","Type":"ContainerStarted","Data":"30a209159416aab33ba5169d211f6ed4de1f757a60ffe3e550c0c08e6cd93fb4"}
Sep 17 20:51:58.205423 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:58.205386    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2rvtb" event={"ID":"0efd83dd-81e7-487a-b68e-2b9197c5f655","Type":"ContainerStarted","Data":"5e0c61cc80b4f6948d20fbac10bcd6cf91a77115889aa3df82a948f2a7756daf"}
Sep 17 20:51:58.205775 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:58.205464    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2rvtb" event={"ID":"0efd83dd-81e7-487a-b68e-2b9197c5f655","Type":"ContainerStarted","Data":"8b0400447e457b7d4fc78a1c917e0a27e5691d9e31ee45aef3b92b7a6b1961da"}
Sep 17 20:51:58.240285 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:51:58.240235    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2rvtb" podStartSLOduration=131.371258105 podStartE2EDuration="2m12.240221506s" totalImagesPullingTime="868.963401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:51:58.2394935 +0000 UTC m=+133.773433309" watchObservedRunningTime="2026-09-17 20:51:58.240221506 +0000 UTC m=+133.774161315"
Sep 17 20:52:02.215446 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:02.215404    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" podUID="f5e84afe-be9d-4e18-96eb-5ce91cfebf15" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 20:52:02.215837 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:02.215477    2586 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk"
Sep 17 20:52:02.215944 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:02.215912    2586 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"13cdd8fe84222edc2ab8adc105e5765a7c1c249bd593a26ad383c936928964fb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 20:52:02.216004 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:02.215972    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" podUID="f5e84afe-be9d-4e18-96eb-5ce91cfebf15" containerName="service-proxy" containerID="cri-o://13cdd8fe84222edc2ab8adc105e5765a7c1c249bd593a26ad383c936928964fb" gracePeriod=30
Sep 17 20:52:03.223541 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:03.223508    2586 generic.go:356] "Generic (PLEG): container finished" podID="f5e84afe-be9d-4e18-96eb-5ce91cfebf15" containerID="13cdd8fe84222edc2ab8adc105e5765a7c1c249bd593a26ad383c936928964fb" exitCode=2
Sep 17 20:52:03.223875 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:03.223575    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" event={"ID":"f5e84afe-be9d-4e18-96eb-5ce91cfebf15","Type":"ContainerDied","Data":"13cdd8fe84222edc2ab8adc105e5765a7c1c249bd593a26ad383c936928964fb"}
Sep 17 20:52:03.223875 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:03.223619    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b55f68ff-898tk" event={"ID":"f5e84afe-be9d-4e18-96eb-5ce91cfebf15","Type":"ContainerStarted","Data":"1619d71a59950c50cd5b0015f85652809b5e28cd2d6159dc8bf7bbdbdf766d8f"}
Sep 17 20:52:16.591725 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.591685    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw" podUID="5d8738bd-241e-49cc-88d3-758ad44cef5d" containerName="registry" containerID="cri-o://c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128" gracePeriod=30
Sep 17 20:52:16.828895 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.828870    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:52:16.932603 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932582    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-bound-sa-token\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.932766 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932616    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-trusted-ca\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.932766 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932648    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvh9z\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-kube-api-access-bvh9z\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.932766 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932673    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d8738bd-241e-49cc-88d3-758ad44cef5d-ca-trust-extracted\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.932766 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932693    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-installation-pull-secrets\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.932766 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932724    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-certificates\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.932766 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932748    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-image-registry-private-configuration\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.932766 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.932766    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") pod \"5d8738bd-241e-49cc-88d3-758ad44cef5d\" (UID: \"5d8738bd-241e-49cc-88d3-758ad44cef5d\") "
Sep 17 20:52:16.933111 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.933059    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:16.933556 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.933448    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 20:52:16.935275 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.935229    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:16.935498 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.935474    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-kube-api-access-bvh9z" (OuterVolumeSpecName: "kube-api-access-bvh9z") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "kube-api-access-bvh9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:16.935589 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.935508    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:52:16.935589 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.935515    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:16.935696 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.935609    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 20:52:16.942420 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:16.942395    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8738bd-241e-49cc-88d3-758ad44cef5d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5d8738bd-241e-49cc-88d3-758ad44cef5d" (UID: "5d8738bd-241e-49cc-88d3-758ad44cef5d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:52:17.033151 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033128    2586 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-certificates\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.033251 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033154    2586 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-image-registry-private-configuration\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.033251 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033170    2586 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-registry-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.033251 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033182    2586 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-bound-sa-token\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.033251 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033194    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d8738bd-241e-49cc-88d3-758ad44cef5d-trusted-ca\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.033251 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033205    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvh9z\" (UniqueName: \"kubernetes.io/projected/5d8738bd-241e-49cc-88d3-758ad44cef5d-kube-api-access-bvh9z\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.033251 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033217    2586 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5d8738bd-241e-49cc-88d3-758ad44cef5d-ca-trust-extracted\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.033251 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.033231    2586 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5d8738bd-241e-49cc-88d3-758ad44cef5d-installation-pull-secrets\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:52:17.267770 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.267697    2586 generic.go:356] "Generic (PLEG): container finished" podID="5d8738bd-241e-49cc-88d3-758ad44cef5d" containerID="c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128" exitCode=0
Sep 17 20:52:17.267770 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.267739    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw"
Sep 17 20:52:17.267770 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.267759    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw" event={"ID":"5d8738bd-241e-49cc-88d3-758ad44cef5d","Type":"ContainerDied","Data":"c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128"}
Sep 17 20:52:17.267951 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.267796    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f479bc4b-mb5qw" event={"ID":"5d8738bd-241e-49cc-88d3-758ad44cef5d","Type":"ContainerDied","Data":"5d70dd160173aaf520ff1b0e36cad9f5e8511084fc688351e0c76027c7d12fa6"}
Sep 17 20:52:17.267951 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.267818    2586 scope.go:117] "RemoveContainer" containerID="c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128"
Sep 17 20:52:17.277039 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.277022    2586 scope.go:117] "RemoveContainer" containerID="c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128"
Sep 17 20:52:17.277291 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:52:17.277273    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128\": container with ID starting with c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128 not found: ID does not exist" containerID="c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128"
Sep 17 20:52:17.277345 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.277297    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128"} err="failed to get container status \"c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128\": rpc error: code = NotFound desc = could not find container \"c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128\": container with ID starting with c2feea71b938ab35b11528f2bdfb5e3cfb791ff5c43e45fe0ae36c572e4b3128 not found: ID does not exist"
Sep 17 20:52:17.293726 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.293703    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f479bc4b-mb5qw"]
Sep 17 20:52:17.298045 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:17.298024    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f479bc4b-mb5qw"]
Sep 17 20:52:18.670310 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:52:18.670277    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d8738bd-241e-49cc-88d3-758ad44cef5d" path="/var/lib/kubelet/pods/5d8738bd-241e-49cc-88d3-758ad44cef5d/volumes"
Sep 17 20:54:46.535198 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:54:46.535166    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:54:46.538425 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:54:46.538402    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:54:46.541848 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:54:46.541821    2586 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 20:56:52.647401 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.647322    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"]
Sep 17 20:56:52.647893 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.647707    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8738bd-241e-49cc-88d3-758ad44cef5d" containerName="registry"
Sep 17 20:56:52.647893 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.647726    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8738bd-241e-49cc-88d3-758ad44cef5d" containerName="registry"
Sep 17 20:56:52.647893 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.647788    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8738bd-241e-49cc-88d3-758ad44cef5d" containerName="registry"
Sep 17 20:56:52.650516 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.650497    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:52.653725 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.653704    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-zcb5q\""
Sep 17 20:56:52.653813 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.653786    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 20:56:52.654519 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.654500    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 20:56:52.654611 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.654536    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 20:56:52.665494 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.665474    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"]
Sep 17 20:56:52.708878 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.708848    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a641e910-30bd-4176-995b-6ec545163e8e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-b8mkq\" (UID: \"a641e910-30bd-4176-995b-6ec545163e8e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:52.709006 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.708884    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnsxf\" (UniqueName: \"kubernetes.io/projected/a641e910-30bd-4176-995b-6ec545163e8e-kube-api-access-wnsxf\") pod \"llmisvc-controller-manager-645f9c8fd6-b8mkq\" (UID: \"a641e910-30bd-4176-995b-6ec545163e8e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:52.810139 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.810112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a641e910-30bd-4176-995b-6ec545163e8e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-b8mkq\" (UID: \"a641e910-30bd-4176-995b-6ec545163e8e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:52.810244 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.810148    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnsxf\" (UniqueName: \"kubernetes.io/projected/a641e910-30bd-4176-995b-6ec545163e8e-kube-api-access-wnsxf\") pod \"llmisvc-controller-manager-645f9c8fd6-b8mkq\" (UID: \"a641e910-30bd-4176-995b-6ec545163e8e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:52.812453 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.812433    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a641e910-30bd-4176-995b-6ec545163e8e-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-b8mkq\" (UID: \"a641e910-30bd-4176-995b-6ec545163e8e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:52.819012 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.818992    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnsxf\" (UniqueName: \"kubernetes.io/projected/a641e910-30bd-4176-995b-6ec545163e8e-kube-api-access-wnsxf\") pod \"llmisvc-controller-manager-645f9c8fd6-b8mkq\" (UID: \"a641e910-30bd-4176-995b-6ec545163e8e\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:52.960241 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:52.960174    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:53.227287 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:53.227216    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"]
Sep 17 20:56:53.230561 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:56:53.230537    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda641e910_30bd_4176_995b_6ec545163e8e.slice/crio-bfef75830e25ab7d5cef49482c0b2bf2d0daf6bd30285054a04b97de39dfa8d6 WatchSource:0}: Error finding container bfef75830e25ab7d5cef49482c0b2bf2d0daf6bd30285054a04b97de39dfa8d6: Status 404 returned error can't find the container with id bfef75830e25ab7d5cef49482c0b2bf2d0daf6bd30285054a04b97de39dfa8d6
Sep 17 20:56:53.231838 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:53.231822    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 20:56:54.133713 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:54.133669    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq" event={"ID":"a641e910-30bd-4176-995b-6ec545163e8e","Type":"ContainerStarted","Data":"bfef75830e25ab7d5cef49482c0b2bf2d0daf6bd30285054a04b97de39dfa8d6"}
Sep 17 20:56:56.143024 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:56.142977    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq" event={"ID":"a641e910-30bd-4176-995b-6ec545163e8e","Type":"ContainerStarted","Data":"179ac82f088a8394d9ea05fa1b7ec687f1080aff056aee27e21a042b074a06dc"}
Sep 17 20:56:56.143407 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:56.143053    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:56:56.159505 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:56:56.159460    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq" podStartSLOduration=1.739483872 podStartE2EDuration="4.159419229s" totalImagesPullingTime="2.419935357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:56:56.158563081 +0000 UTC m=+431.692502891" watchObservedRunningTime="2026-09-17 20:56:56.159419229 +0000 UTC m=+431.693359037"
Sep 17 20:57:27.148822 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:57:27.148788    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-b8mkq"
Sep 17 20:58:14.464411 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.464381    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-qds2c"]
Sep 17 20:58:14.467382 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.467346    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.473620 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.473592    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 20:58:14.473724 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.473631    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-nwnrz\""
Sep 17 20:58:14.474070 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.474055    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 20:58:14.489208 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.489177    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-qds2c"]
Sep 17 20:58:14.550091 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.550063    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-metrics-certs\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.550208 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.550100    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-cert\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.550208 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.550146    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5kd\" (UniqueName: \"kubernetes.io/projected/869396f8-52cc-46b9-a014-8f625e0311bf-kube-api-access-cx5kd\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.651161 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.651134    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5kd\" (UniqueName: \"kubernetes.io/projected/869396f8-52cc-46b9-a014-8f625e0311bf-kube-api-access-cx5kd\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.651285 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.651171    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-metrics-certs\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.651285 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.651193    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-cert\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.651442 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:58:14.651282    2586 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 17 20:58:14.651442 ip-10-0-143-254 kubenswrapper[2586]: E0917 20:58:14.651346    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-metrics-certs podName:869396f8-52cc-46b9-a014-8f625e0311bf nodeName:}" failed. No retries permitted until 2026-09-17 20:58:15.151330895 +0000 UTC m=+510.685270681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-metrics-certs") pod "odh-model-controller-7b966d7984-qds2c" (UID: "869396f8-52cc-46b9-a014-8f625e0311bf") : secret "odh-model-controller-metrics-tls" not found
Sep 17 20:58:14.653649 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.653628    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-cert\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:14.660057 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:14.660037    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5kd\" (UniqueName: \"kubernetes.io/projected/869396f8-52cc-46b9-a014-8f625e0311bf-kube-api-access-cx5kd\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:15.153406 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:15.153374    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-metrics-certs\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:15.155683 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:15.155655    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/869396f8-52cc-46b9-a014-8f625e0311bf-metrics-certs\") pod \"odh-model-controller-7b966d7984-qds2c\" (UID: \"869396f8-52cc-46b9-a014-8f625e0311bf\") " pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:15.376564 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:15.376517    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:15.648132 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:15.645969    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-qds2c"]
Sep 17 20:58:16.404674 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:16.404620    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-qds2c" event={"ID":"869396f8-52cc-46b9-a014-8f625e0311bf","Type":"ContainerStarted","Data":"4b3f72b42033cb309b9eda4957fc67bc3a2196bff308151e54be64fd02479aa8"}
Sep 17 20:58:19.418286 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:19.418251    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-qds2c" event={"ID":"869396f8-52cc-46b9-a014-8f625e0311bf","Type":"ContainerStarted","Data":"e396ebd73ae8b4108b29d3b6e18d88cb317c6952788007bea7d45837b91b1f8a"}
Sep 17 20:58:19.418712 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:19.418404    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:19.436101 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:19.436048    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-qds2c" podStartSLOduration=2.397116263 podStartE2EDuration="5.436032989s" totalImagesPullingTime="3.038916726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:19.434623917 +0000 UTC m=+514.968563725" watchObservedRunningTime="2026-09-17 20:58:19.436032989 +0000 UTC m=+514.969972800"
Sep 17 20:58:30.424144 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:30.424114    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-qds2c"
Sep 17 20:58:31.215870 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.215839    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-st85t"]
Sep 17 20:58:31.218596 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.218577    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-st85t"
Sep 17 20:58:31.220965 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.220941    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 20:58:31.220965 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.220960    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 20:58:31.221122 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.220960    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d2mrz\""
Sep 17 20:58:31.227072 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.227051    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-st85t"]
Sep 17 20:58:31.368337 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.368311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlqn\" (UniqueName: \"kubernetes.io/projected/a5913159-3c07-4321-a774-e6ac985e54df-kube-api-access-pdlqn\") pod \"s3-init-st85t\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") " pod="kserve/s3-init-st85t"
Sep 17 20:58:31.368490 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.368426    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5913159-3c07-4321-a774-e6ac985e54df-model-cache\") pod \"s3-init-st85t\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") " pod="kserve/s3-init-st85t"
Sep 17 20:58:31.469523 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.469450    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5913159-3c07-4321-a774-e6ac985e54df-model-cache\") pod \"s3-init-st85t\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") " pod="kserve/s3-init-st85t"
Sep 17 20:58:31.469523 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.469486    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlqn\" (UniqueName: \"kubernetes.io/projected/a5913159-3c07-4321-a774-e6ac985e54df-kube-api-access-pdlqn\") pod \"s3-init-st85t\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") " pod="kserve/s3-init-st85t"
Sep 17 20:58:31.469887 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.469800    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5913159-3c07-4321-a774-e6ac985e54df-model-cache\") pod \"s3-init-st85t\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") " pod="kserve/s3-init-st85t"
Sep 17 20:58:31.477919 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.477897    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlqn\" (UniqueName: \"kubernetes.io/projected/a5913159-3c07-4321-a774-e6ac985e54df-kube-api-access-pdlqn\") pod \"s3-init-st85t\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") " pod="kserve/s3-init-st85t"
Sep 17 20:58:31.527156 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.527131    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-st85t"
Sep 17 20:58:31.795480 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:31.795433    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-st85t"]
Sep 17 20:58:31.798437 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:58:31.798408    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5913159_3c07_4321_a774_e6ac985e54df.slice/crio-370275c106bdd26c0f84132302159cb0058bc8e0ff5f88d2400abd3d2b033405 WatchSource:0}: Error finding container 370275c106bdd26c0f84132302159cb0058bc8e0ff5f88d2400abd3d2b033405: Status 404 returned error can't find the container with id 370275c106bdd26c0f84132302159cb0058bc8e0ff5f88d2400abd3d2b033405
Sep 17 20:58:32.462238 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:32.462196    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-st85t" event={"ID":"a5913159-3c07-4321-a774-e6ac985e54df","Type":"ContainerStarted","Data":"370275c106bdd26c0f84132302159cb0058bc8e0ff5f88d2400abd3d2b033405"}
Sep 17 20:58:36.478608 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:36.478571    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-st85t" event={"ID":"a5913159-3c07-4321-a774-e6ac985e54df","Type":"ContainerStarted","Data":"95e8995ba4067ad151eaa3686ded901660f1670cbd8c87a8bfd6d6ac292aed88"}
Sep 17 20:58:37.483968 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:37.483937    2586 generic.go:356] "Generic (PLEG): container finished" podID="a5913159-3c07-4321-a774-e6ac985e54df" containerID="95e8995ba4067ad151eaa3686ded901660f1670cbd8c87a8bfd6d6ac292aed88" exitCode=0
Sep 17 20:58:37.484335 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:37.484039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-st85t" event={"ID":"a5913159-3c07-4321-a774-e6ac985e54df","Type":"ContainerDied","Data":"95e8995ba4067ad151eaa3686ded901660f1670cbd8c87a8bfd6d6ac292aed88"}
Sep 17 20:58:42.506095 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:42.506052    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-st85t" event={"ID":"a5913159-3c07-4321-a774-e6ac985e54df","Type":"ContainerStarted","Data":"13bb2ddd72963e5520ac2a845f8c016ef7aef55fdafb2347e3a6af2f9c97821e"}
Sep 17 20:58:42.536272 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:58:42.536227    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-st85t" podStartSLOduration=1.790539759 podStartE2EDuration="11.536210888s" totalImagesPullingTime="7.992109593s" totalInitContainerRuntime="1.753561536s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:58:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:58:42.536101577 +0000 UTC m=+538.070041387" watchObservedRunningTime="2026-09-17 20:58:42.536210888 +0000 UTC m=+538.070150695"
Sep 17 20:59:46.555700 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:46.555668    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:59:46.558644 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:46.558623    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 20:59:46.719998 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:46.719961    2586 generic.go:356] "Generic (PLEG): container finished" podID="a5913159-3c07-4321-a774-e6ac985e54df" containerID="13bb2ddd72963e5520ac2a845f8c016ef7aef55fdafb2347e3a6af2f9c97821e" exitCode=0
Sep 17 20:59:46.720152 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:46.720028    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-st85t" event={"ID":"a5913159-3c07-4321-a774-e6ac985e54df","Type":"ContainerDied","Data":"13bb2ddd72963e5520ac2a845f8c016ef7aef55fdafb2347e3a6af2f9c97821e"}
Sep 17 20:59:47.840598 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:47.840577    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-st85t"
Sep 17 20:59:47.869935 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:47.869906    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdlqn\" (UniqueName: \"kubernetes.io/projected/a5913159-3c07-4321-a774-e6ac985e54df-kube-api-access-pdlqn\") pod \"a5913159-3c07-4321-a774-e6ac985e54df\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") "
Sep 17 20:59:47.870064 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:47.869946    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5913159-3c07-4321-a774-e6ac985e54df-model-cache\") pod \"a5913159-3c07-4321-a774-e6ac985e54df\" (UID: \"a5913159-3c07-4321-a774-e6ac985e54df\") "
Sep 17 20:59:47.871963 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:47.871939    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5913159-3c07-4321-a774-e6ac985e54df-kube-api-access-pdlqn" (OuterVolumeSpecName: "kube-api-access-pdlqn") pod "a5913159-3c07-4321-a774-e6ac985e54df" (UID: "a5913159-3c07-4321-a774-e6ac985e54df"). InnerVolumeSpecName "kube-api-access-pdlqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 20:59:47.888773 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:47.888745    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5913159-3c07-4321-a774-e6ac985e54df-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5913159-3c07-4321-a774-e6ac985e54df" (UID: "a5913159-3c07-4321-a774-e6ac985e54df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 20:59:47.970619 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:47.970599    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5913159-3c07-4321-a774-e6ac985e54df-model-cache\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:59:47.970686 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:47.970620    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlqn\" (UniqueName: \"kubernetes.io/projected/a5913159-3c07-4321-a774-e6ac985e54df-kube-api-access-pdlqn\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 20:59:48.727710 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:48.727675    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-st85t" event={"ID":"a5913159-3c07-4321-a774-e6ac985e54df","Type":"ContainerDied","Data":"370275c106bdd26c0f84132302159cb0058bc8e0ff5f88d2400abd3d2b033405"}
Sep 17 20:59:48.727710 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:48.727712    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="370275c106bdd26c0f84132302159cb0058bc8e0ff5f88d2400abd3d2b033405"
Sep 17 20:59:48.727863 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:48.727735    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-st85t"
Sep 17 20:59:55.784219 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.784187    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-7hw5m"]
Sep 17 20:59:55.784612 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.784467    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5913159-3c07-4321-a774-e6ac985e54df" containerName="download-hf-model"
Sep 17 20:59:55.784612 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.784479    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5913159-3c07-4321-a774-e6ac985e54df" containerName="download-hf-model"
Sep 17 20:59:55.784612 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.784487    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5913159-3c07-4321-a774-e6ac985e54df" containerName="s3-init"
Sep 17 20:59:55.784612 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.784492    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5913159-3c07-4321-a774-e6ac985e54df" containerName="s3-init"
Sep 17 20:59:55.784612 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.784532    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5913159-3c07-4321-a774-e6ac985e54df" containerName="s3-init"
Sep 17 20:59:55.787698 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.787682    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:55.789872 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.789853    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 20:59:55.789872 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.789860    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 20:59:55.790062 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.789949    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d2mrz\""
Sep 17 20:59:55.793842 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.793819    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7hw5m"]
Sep 17 20:59:55.821336 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.821311    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7z6\" (UniqueName: \"kubernetes.io/projected/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-kube-api-access-mv7z6\") pod \"s3-tls-init-custom-7hw5m\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") " pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:55.821458 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.821346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-model-cache\") pod \"s3-tls-init-custom-7hw5m\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") " pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:55.921687 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.921661    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7z6\" (UniqueName: \"kubernetes.io/projected/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-kube-api-access-mv7z6\") pod \"s3-tls-init-custom-7hw5m\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") " pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:55.921784 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.921694    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-model-cache\") pod \"s3-tls-init-custom-7hw5m\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") " pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:55.922056 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.922037    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-model-cache\") pod \"s3-tls-init-custom-7hw5m\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") " pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:55.929615 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:55.929589    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7z6\" (UniqueName: \"kubernetes.io/projected/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-kube-api-access-mv7z6\") pod \"s3-tls-init-custom-7hw5m\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") " pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:56.096628 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:56.096542    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 20:59:56.373050 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:56.373021    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7hw5m"]
Sep 17 20:59:56.377452 ip-10-0-143-254 kubenswrapper[2586]: W0917 20:59:56.377424    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb79a5af_bebd_40eb_8106_9ce2233eb6d2.slice/crio-fa492fddeae6f11d34de51fd7c6a8610ee26bfff8d9547a5a7a287461c41ae31 WatchSource:0}: Error finding container fa492fddeae6f11d34de51fd7c6a8610ee26bfff8d9547a5a7a287461c41ae31: Status 404 returned error can't find the container with id fa492fddeae6f11d34de51fd7c6a8610ee26bfff8d9547a5a7a287461c41ae31
Sep 17 20:59:56.760202 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:56.760122    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7hw5m" event={"ID":"bb79a5af-bebd-40eb-8106-9ce2233eb6d2","Type":"ContainerStarted","Data":"4d37b5b4d86c4ff4250541019e854c65480d504a6be14a2377ff7b88a0f0a35e"}
Sep 17 20:59:56.760202 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:56.760163    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7hw5m" event={"ID":"bb79a5af-bebd-40eb-8106-9ce2233eb6d2","Type":"ContainerStarted","Data":"fa492fddeae6f11d34de51fd7c6a8610ee26bfff8d9547a5a7a287461c41ae31"}
Sep 17 20:59:58.769669 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:58.769639    2586 generic.go:356] "Generic (PLEG): container finished" podID="bb79a5af-bebd-40eb-8106-9ce2233eb6d2" containerID="4d37b5b4d86c4ff4250541019e854c65480d504a6be14a2377ff7b88a0f0a35e" exitCode=0
Sep 17 20:59:58.770077 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:58.769701    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7hw5m" event={"ID":"bb79a5af-bebd-40eb-8106-9ce2233eb6d2","Type":"ContainerDied","Data":"4d37b5b4d86c4ff4250541019e854c65480d504a6be14a2377ff7b88a0f0a35e"}
Sep 17 20:59:59.774453 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:59.774415    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7hw5m" event={"ID":"bb79a5af-bebd-40eb-8106-9ce2233eb6d2","Type":"ContainerStarted","Data":"50b8c79e2895872c5c4974df3f4bf8ba30c60a16b90ef9c2fdd28a8a6bb17767"}
Sep 17 20:59:59.793894 ip-10-0-143-254 kubenswrapper[2586]: I0917 20:59:59.793843    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-7hw5m" podStartSLOduration=3.062293438 podStartE2EDuration="4.793829018s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.73153558s" isStatefulPod=false podCreationTimestamp="2026-09-17 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:59:59.79299954 +0000 UTC m=+615.326939349" watchObservedRunningTime="2026-09-17 20:59:59.793829018 +0000 UTC m=+615.327768827"
Sep 17 21:00:05.796435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:05.796400    2586 generic.go:356] "Generic (PLEG): container finished" podID="bb79a5af-bebd-40eb-8106-9ce2233eb6d2" containerID="50b8c79e2895872c5c4974df3f4bf8ba30c60a16b90ef9c2fdd28a8a6bb17767" exitCode=0
Sep 17 21:00:05.796801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:05.796443    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7hw5m" event={"ID":"bb79a5af-bebd-40eb-8106-9ce2233eb6d2","Type":"ContainerDied","Data":"50b8c79e2895872c5c4974df3f4bf8ba30c60a16b90ef9c2fdd28a8a6bb17767"}
Sep 17 21:00:06.917631 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:06.917609    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 21:00:07.004040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.004014    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-model-cache\") pod \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") "
Sep 17 21:00:07.004180 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.004080    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv7z6\" (UniqueName: \"kubernetes.io/projected/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-kube-api-access-mv7z6\") pod \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\" (UID: \"bb79a5af-bebd-40eb-8106-9ce2233eb6d2\") "
Sep 17 21:00:07.006073 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.006038    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-kube-api-access-mv7z6" (OuterVolumeSpecName: "kube-api-access-mv7z6") pod "bb79a5af-bebd-40eb-8106-9ce2233eb6d2" (UID: "bb79a5af-bebd-40eb-8106-9ce2233eb6d2"). InnerVolumeSpecName "kube-api-access-mv7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:07.026790 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.026764    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb79a5af-bebd-40eb-8106-9ce2233eb6d2" (UID: "bb79a5af-bebd-40eb-8106-9ce2233eb6d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:00:07.104865 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.104811    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7z6\" (UniqueName: \"kubernetes.io/projected/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-kube-api-access-mv7z6\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:07.104865 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.104833    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb79a5af-bebd-40eb-8106-9ce2233eb6d2-model-cache\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:07.804035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.804002    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7hw5m" event={"ID":"bb79a5af-bebd-40eb-8106-9ce2233eb6d2","Type":"ContainerDied","Data":"fa492fddeae6f11d34de51fd7c6a8610ee26bfff8d9547a5a7a287461c41ae31"}
Sep 17 21:00:07.804035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.804038    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa492fddeae6f11d34de51fd7c6a8610ee26bfff8d9547a5a7a287461c41ae31"
Sep 17 21:00:07.804273 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:07.804042    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7hw5m"
Sep 17 21:00:10.118017 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.117985    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-46hb8"]
Sep 17 21:00:10.118424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.118204    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb79a5af-bebd-40eb-8106-9ce2233eb6d2" containerName="download-hf-model"
Sep 17 21:00:10.118424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.118214    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb79a5af-bebd-40eb-8106-9ce2233eb6d2" containerName="download-hf-model"
Sep 17 21:00:10.118424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.118222    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb79a5af-bebd-40eb-8106-9ce2233eb6d2" containerName="s3-tls-init-custom"
Sep 17 21:00:10.118424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.118228    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb79a5af-bebd-40eb-8106-9ce2233eb6d2" containerName="s3-tls-init-custom"
Sep 17 21:00:10.118424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.118270    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb79a5af-bebd-40eb-8106-9ce2233eb6d2" containerName="s3-tls-init-custom"
Sep 17 21:00:10.120930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.120914    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.123320 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.123299    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 21:00:10.123444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.123299    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 21:00:10.123444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.123339    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d2mrz\""
Sep 17 21:00:10.130226 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.130203    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-46hb8"]
Sep 17 21:00:10.223495 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.223464    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqhk\" (UniqueName: \"kubernetes.io/projected/85de3a6f-cc0e-4ebe-9ece-c109813ba960-kube-api-access-jgqhk\") pod \"s3-tls-init-serving-46hb8\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") " pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.223655 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.223522    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de3a6f-cc0e-4ebe-9ece-c109813ba960-model-cache\") pod \"s3-tls-init-serving-46hb8\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") " pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.324780 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.324746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqhk\" (UniqueName: \"kubernetes.io/projected/85de3a6f-cc0e-4ebe-9ece-c109813ba960-kube-api-access-jgqhk\") pod \"s3-tls-init-serving-46hb8\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") " pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.324922 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.324803    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de3a6f-cc0e-4ebe-9ece-c109813ba960-model-cache\") pod \"s3-tls-init-serving-46hb8\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") " pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.325161 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.325145    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de3a6f-cc0e-4ebe-9ece-c109813ba960-model-cache\") pod \"s3-tls-init-serving-46hb8\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") " pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.337305 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.337272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqhk\" (UniqueName: \"kubernetes.io/projected/85de3a6f-cc0e-4ebe-9ece-c109813ba960-kube-api-access-jgqhk\") pod \"s3-tls-init-serving-46hb8\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") " pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.430178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.430109    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:10.699833 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.699809    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-46hb8"]
Sep 17 21:00:10.702262 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:00:10.702235    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85de3a6f_cc0e_4ebe_9ece_c109813ba960.slice/crio-56044cdf2ebb23b78c069ac870cd5d0a3f6b9ea78d742b1c72d2e1c98fc00006 WatchSource:0}: Error finding container 56044cdf2ebb23b78c069ac870cd5d0a3f6b9ea78d742b1c72d2e1c98fc00006: Status 404 returned error can't find the container with id 56044cdf2ebb23b78c069ac870cd5d0a3f6b9ea78d742b1c72d2e1c98fc00006
Sep 17 21:00:10.816556 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.816520    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-46hb8" event={"ID":"85de3a6f-cc0e-4ebe-9ece-c109813ba960","Type":"ContainerStarted","Data":"c490b87dd9c39af0ca6307bc043eb074f6dbf866dc7c091eec62385d3f300732"}
Sep 17 21:00:10.816708 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:10.816564    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-46hb8" event={"ID":"85de3a6f-cc0e-4ebe-9ece-c109813ba960","Type":"ContainerStarted","Data":"56044cdf2ebb23b78c069ac870cd5d0a3f6b9ea78d742b1c72d2e1c98fc00006"}
Sep 17 21:00:12.825820 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:12.825787    2586 generic.go:356] "Generic (PLEG): container finished" podID="85de3a6f-cc0e-4ebe-9ece-c109813ba960" containerID="c490b87dd9c39af0ca6307bc043eb074f6dbf866dc7c091eec62385d3f300732" exitCode=0
Sep 17 21:00:12.826179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:12.825829    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-46hb8" event={"ID":"85de3a6f-cc0e-4ebe-9ece-c109813ba960","Type":"ContainerDied","Data":"c490b87dd9c39af0ca6307bc043eb074f6dbf866dc7c091eec62385d3f300732"}
Sep 17 21:00:13.831622 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:13.831581    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-46hb8" event={"ID":"85de3a6f-cc0e-4ebe-9ece-c109813ba960","Type":"ContainerStarted","Data":"0244fa301b0a54c4fb0ed7b81a467cdbd135f1cc1c21cd3782b59c6ec11518d2"}
Sep 17 21:00:13.853543 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:13.853500    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-46hb8" podStartSLOduration=2.299138603 podStartE2EDuration="3.853484517s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.554345914s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:13.852444276 +0000 UTC m=+629.386384097" watchObservedRunningTime="2026-09-17 21:00:13.853484517 +0000 UTC m=+629.387424327"
Sep 17 21:00:18.856629 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:18.856600    2586 generic.go:356] "Generic (PLEG): container finished" podID="85de3a6f-cc0e-4ebe-9ece-c109813ba960" containerID="0244fa301b0a54c4fb0ed7b81a467cdbd135f1cc1c21cd3782b59c6ec11518d2" exitCode=0
Sep 17 21:00:18.856940 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:18.856668    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-46hb8" event={"ID":"85de3a6f-cc0e-4ebe-9ece-c109813ba960","Type":"ContainerDied","Data":"0244fa301b0a54c4fb0ed7b81a467cdbd135f1cc1c21cd3782b59c6ec11518d2"}
Sep 17 21:00:19.985470 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:19.985449    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:20.089962 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.089929    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgqhk\" (UniqueName: \"kubernetes.io/projected/85de3a6f-cc0e-4ebe-9ece-c109813ba960-kube-api-access-jgqhk\") pod \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") "
Sep 17 21:00:20.089962 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.089969    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de3a6f-cc0e-4ebe-9ece-c109813ba960-model-cache\") pod \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\" (UID: \"85de3a6f-cc0e-4ebe-9ece-c109813ba960\") "
Sep 17 21:00:20.091990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.091955    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85de3a6f-cc0e-4ebe-9ece-c109813ba960-kube-api-access-jgqhk" (OuterVolumeSpecName: "kube-api-access-jgqhk") pod "85de3a6f-cc0e-4ebe-9ece-c109813ba960" (UID: "85de3a6f-cc0e-4ebe-9ece-c109813ba960"). InnerVolumeSpecName "kube-api-access-jgqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:20.113242 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.113187    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85de3a6f-cc0e-4ebe-9ece-c109813ba960-model-cache" (OuterVolumeSpecName: "model-cache") pod "85de3a6f-cc0e-4ebe-9ece-c109813ba960" (UID: "85de3a6f-cc0e-4ebe-9ece-c109813ba960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:00:20.190350 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.190324    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgqhk\" (UniqueName: \"kubernetes.io/projected/85de3a6f-cc0e-4ebe-9ece-c109813ba960-kube-api-access-jgqhk\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.190350 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.190350    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85de3a6f-cc0e-4ebe-9ece-c109813ba960-model-cache\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:20.865123 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.865093    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-46hb8" event={"ID":"85de3a6f-cc0e-4ebe-9ece-c109813ba960","Type":"ContainerDied","Data":"56044cdf2ebb23b78c069ac870cd5d0a3f6b9ea78d742b1c72d2e1c98fc00006"}
Sep 17 21:00:20.865123 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.865128    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56044cdf2ebb23b78c069ac870cd5d0a3f6b9ea78d742b1c72d2e1c98fc00006"
Sep 17 21:00:20.865315 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:20.865126    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-46hb8"
Sep 17 21:00:33.869973 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.869943    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"]
Sep 17 21:00:33.871270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.870165    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85de3a6f-cc0e-4ebe-9ece-c109813ba960" containerName="download-hf-model"
Sep 17 21:00:33.871270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.870175    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="85de3a6f-cc0e-4ebe-9ece-c109813ba960" containerName="download-hf-model"
Sep 17 21:00:33.871270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.870191    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85de3a6f-cc0e-4ebe-9ece-c109813ba960" containerName="s3-tls-init-serving"
Sep 17 21:00:33.871270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.870197    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="85de3a6f-cc0e-4ebe-9ece-c109813ba960" containerName="s3-tls-init-serving"
Sep 17 21:00:33.871270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.870244    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="85de3a6f-cc0e-4ebe-9ece-c109813ba960" containerName="s3-tls-init-serving"
Sep 17 21:00:33.872218 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.872203    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:33.874612 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.874590    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 21:00:33.874730 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.874710    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:00:33.874821 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.874806    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:00:33.874884 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.874830    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:00:33.875573 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.875556    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:00:33.875696 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.875672    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:00:33.875742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.875674    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 21:00:33.883014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.882995    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"]
Sep 17 21:00:33.979186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.979160    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:33.979334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.979200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272ec759-4267-4ae4-aeb0-cd3ac5382a64-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:33.979334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.979222    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:33.979334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.979247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:33.979334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:33.979285    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndhxw\" (UniqueName: \"kubernetes.io/projected/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-api-access-ndhxw\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.040244 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.040210    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"]
Sep 17 21:00:34.042764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.042748    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.045255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.045234    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-58902f-predictor-serving-cert\""
Sep 17 21:00:34.045390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.045254    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-58902f-dockercfg-hjk6x\""
Sep 17 21:00:34.045390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.045276    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-58902f\""
Sep 17 21:00:34.045390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.045303    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-58902f-kube-rbac-proxy-sar-config\""
Sep 17 21:00:34.054071 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.054047    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"]
Sep 17 21:00:34.080606 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.080579    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.080739 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.080621    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272ec759-4267-4ae4-aeb0-cd3ac5382a64-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.080739 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.080643    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.080739 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.080658    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.080739 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.080675    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndhxw\" (UniqueName: \"kubernetes.io/projected/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-api-access-ndhxw\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.081055 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.081036    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.081453 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.081432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.081587 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.081572    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.083112 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.083087    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272ec759-4267-4ae4-aeb0-cd3ac5382a64-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.087950 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.087930    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndhxw\" (UniqueName: \"kubernetes.io/projected/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-api-access-ndhxw\") pod \"isvc-sklearn-batcher-predictor-77774898d8-sct72\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.181706 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.181644    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.181706 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.181673    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kserve-provision-location\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.181706 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.181705    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cm6\" (UniqueName: \"kubernetes.io/projected/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-api-access-h8cm6\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.181871 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.181809    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.181871 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.181826    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-cabundle-cert\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.181871 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.181828    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:00:34.282264 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.282264 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282264    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-cabundle-cert\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.282495 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282292    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.282495 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282317    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kserve-provision-location\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.282495 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282366    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cm6\" (UniqueName: \"kubernetes.io/projected/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-api-access-h8cm6\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.282495 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:00:34.282479    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-serving-cert: secret "isvc-init-fail-58902f-predictor-serving-cert" not found
Sep 17 21:00:34.282729 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:00:34.282551    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls podName:7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0 nodeName:}" failed. No retries permitted until 2026-09-17 21:00:34.782530051 +0000 UTC m=+650.316469839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls") pod "isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" (UID: "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0") : secret "isvc-init-fail-58902f-predictor-serving-cert" not found
Sep 17 21:00:34.282796 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282733    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kserve-provision-location\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.282878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282860    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.282958 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.282940    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-cabundle-cert\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.290451 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.290425    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cm6\" (UniqueName: \"kubernetes.io/projected/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-api-access-h8cm6\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.455268 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.455211    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"]
Sep 17 21:00:34.457535 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:00:34.457509    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272ec759_4267_4ae4_aeb0_cd3ac5382a64.slice/crio-8c24d54cb934d97714f37b7fe53052fdc90a3f3511ba53e2258fd024b124c826 WatchSource:0}: Error finding container 8c24d54cb934d97714f37b7fe53052fdc90a3f3511ba53e2258fd024b124c826: Status 404 returned error can't find the container with id 8c24d54cb934d97714f37b7fe53052fdc90a3f3511ba53e2258fd024b124c826
Sep 17 21:00:34.786608 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.786528    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.788784 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.788767    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls\") pod \"isvc-init-fail-58902f-predictor-788b79c64d-8qvcj\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") " pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:34.916700 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.916663    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerStarted","Data":"aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6"}
Sep 17 21:00:34.917045 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.916741    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerStarted","Data":"8c24d54cb934d97714f37b7fe53052fdc90a3f3511ba53e2258fd024b124c826"}
Sep 17 21:00:34.958134 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:34.958097    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:35.228131 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:35.228084    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"]
Sep 17 21:00:35.230862 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:00:35.230835    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca0cdf0_f8c7_4713_9830_1cc7b3b55ec0.slice/crio-60d735aa798a85f0717fa9127f3cbecb19b57003b3b432676723c05098c7df1b WatchSource:0}: Error finding container 60d735aa798a85f0717fa9127f3cbecb19b57003b3b432676723c05098c7df1b: Status 404 returned error can't find the container with id 60d735aa798a85f0717fa9127f3cbecb19b57003b3b432676723c05098c7df1b
Sep 17 21:00:35.922229 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:35.922187    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" event={"ID":"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0","Type":"ContainerStarted","Data":"017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09"}
Sep 17 21:00:35.922704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:35.922274    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" event={"ID":"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0","Type":"ContainerStarted","Data":"60d735aa798a85f0717fa9127f3cbecb19b57003b3b432676723c05098c7df1b"}
Sep 17 21:00:38.934886 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:38.934854    2586 generic.go:356] "Generic (PLEG): container finished" podID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerID="aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6" exitCode=0
Sep 17 21:00:38.935262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:38.934926    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerDied","Data":"aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6"}
Sep 17 21:00:39.949415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:39.948746    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-58902f-predictor-788b79c64d-8qvcj_7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0/storage-initializer/0.log"
Sep 17 21:00:39.949415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:39.948786    2586 generic.go:356] "Generic (PLEG): container finished" podID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerID="017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09" exitCode=1
Sep 17 21:00:39.949415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:39.948894    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" event={"ID":"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0","Type":"ContainerDied","Data":"017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09"}
Sep 17 21:00:40.958533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:40.957647    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-58902f-predictor-788b79c64d-8qvcj_7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0/storage-initializer/0.log"
Sep 17 21:00:40.958533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:40.957747    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" event={"ID":"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0","Type":"ContainerStarted","Data":"3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f"}
Sep 17 21:00:43.771165 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:43.770251    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"]
Sep 17 21:00:43.771165 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:43.770542    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerName="storage-initializer" containerID="cri-o://3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f" gracePeriod=30
Sep 17 21:00:44.556844 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.556816    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-58902f-predictor-788b79c64d-8qvcj_7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0/storage-initializer/1.log"
Sep 17 21:00:44.557281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.557266    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-58902f-predictor-788b79c64d-8qvcj_7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0/storage-initializer/0.log"
Sep 17 21:00:44.559699 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.559679    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:44.684395 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684315    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-rbac-proxy-sar-config\") pod \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") "
Sep 17 21:00:44.684546 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684393    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kserve-provision-location\") pod \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") "
Sep 17 21:00:44.684546 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684420    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-cabundle-cert\") pod \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") "
Sep 17 21:00:44.684546 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684448    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls\") pod \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") "
Sep 17 21:00:44.684546 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684501    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8cm6\" (UniqueName: \"kubernetes.io/projected/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-api-access-h8cm6\") pod \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\" (UID: \"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0\") "
Sep 17 21:00:44.684744 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684685    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" (UID: "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:00:44.684792 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684716    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" (UID: "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:00:44.684874 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.684838    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" (UID: "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:00:44.686638 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.686616    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" (UID: "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:00:44.686763 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.686746    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-api-access-h8cm6" (OuterVolumeSpecName: "kube-api-access-h8cm6") pod "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" (UID: "7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0"). InnerVolumeSpecName "kube-api-access-h8cm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:00:44.785950 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.785914    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:44.785950 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.785946    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:44.785950 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.785958    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:44.786475 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.785966    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8cm6\" (UniqueName: \"kubernetes.io/projected/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-api-access-h8cm6\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:44.786475 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.785974    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:00:44.978632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.978552    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-58902f-predictor-788b79c64d-8qvcj_7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0/storage-initializer/1.log"
Sep 17 21:00:44.978987 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.978971    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-58902f-predictor-788b79c64d-8qvcj_7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0/storage-initializer/0.log"
Sep 17 21:00:44.979041 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.979006    2586 generic.go:356] "Generic (PLEG): container finished" podID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerID="3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f" exitCode=1
Sep 17 21:00:44.979074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.979039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" event={"ID":"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0","Type":"ContainerDied","Data":"3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f"}
Sep 17 21:00:44.979112 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.979076    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"
Sep 17 21:00:44.979141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.979120    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj" event={"ID":"7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0","Type":"ContainerDied","Data":"60d735aa798a85f0717fa9127f3cbecb19b57003b3b432676723c05098c7df1b"}
Sep 17 21:00:44.979176 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.979157    2586 scope.go:117] "RemoveContainer" containerID="3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f"
Sep 17 21:00:44.988624 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.988604    2586 scope.go:117] "RemoveContainer" containerID="017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09"
Sep 17 21:00:44.996800 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.996783    2586 scope.go:117] "RemoveContainer" containerID="3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f"
Sep 17 21:00:44.997081 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:00:44.997056    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f\": container with ID starting with 3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f not found: ID does not exist" containerID="3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f"
Sep 17 21:00:44.997153 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.997092    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f"} err="failed to get container status \"3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f\": rpc error: code = NotFound desc = could not find container \"3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f\": container with ID starting with 3ca8efd80e2363d11bfd6708dbe82cd69e3253bb7bfcde2368576a92a1666c7f not found: ID does not exist"
Sep 17 21:00:44.997153 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.997114    2586 scope.go:117] "RemoveContainer" containerID="017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09"
Sep 17 21:00:44.997439 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:00:44.997417    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09\": container with ID starting with 017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09 not found: ID does not exist" containerID="017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09"
Sep 17 21:00:44.997510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:44.997448    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09"} err="failed to get container status \"017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09\": rpc error: code = NotFound desc = could not find container \"017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09\": container with ID starting with 017bf9c02666c665e9a6a6c7a5620e1bc2b6dd6bddfe8b4f4727d55dfb8a5e09 not found: ID does not exist"
Sep 17 21:00:45.032303 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:45.032273    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"]
Sep 17 21:00:45.047228 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:45.047198    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-58902f-predictor-788b79c64d-8qvcj"]
Sep 17 21:00:46.671569 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:46.671542    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" path="/var/lib/kubelet/pods/7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0/volumes"
Sep 17 21:00:48.941103 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.941072    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"]
Sep 17 21:00:48.941565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.941439    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerName="storage-initializer"
Sep 17 21:00:48.941565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.941455    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerName="storage-initializer"
Sep 17 21:00:48.941565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.941466    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerName="storage-initializer"
Sep 17 21:00:48.941565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.941474    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerName="storage-initializer"
Sep 17 21:00:48.941565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.941536    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerName="storage-initializer"
Sep 17 21:00:48.941822 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.941653    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca0cdf0-f8c7-4713-9830-1cc7b3b55ec0" containerName="storage-initializer"
Sep 17 21:00:48.955464 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.955432    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:48.959440 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.959413    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 21:00:48.959984 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.959927    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 21:00:48.962803 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:48.962586    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"]
Sep 17 21:00:49.121987 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.121950    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.122163 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.122001    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg2kc\" (UniqueName: \"kubernetes.io/projected/a444ff21-4251-4065-b64d-cf4b89173089-kube-api-access-vg2kc\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.122163 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.122035    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a444ff21-4251-4065-b64d-cf4b89173089-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.122163 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.122059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.122378 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.122210    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a444ff21-4251-4065-b64d-cf4b89173089-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.223577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.223488    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.223577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.223542    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg2kc\" (UniqueName: \"kubernetes.io/projected/a444ff21-4251-4065-b64d-cf4b89173089-kube-api-access-vg2kc\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.223787 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.223706    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a444ff21-4251-4065-b64d-cf4b89173089-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.223787 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.223746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.223890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.223829    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a444ff21-4251-4065-b64d-cf4b89173089-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.224169 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.224148    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a444ff21-4251-4065-b64d-cf4b89173089-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.224296 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.224273    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.224516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.224494    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.226406 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.226388    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a444ff21-4251-4065-b64d-cf4b89173089-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.232158 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.232132    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg2kc\" (UniqueName: \"kubernetes.io/projected/a444ff21-4251-4065-b64d-cf4b89173089-kube-api-access-vg2kc\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-8jg8k\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:49.267984 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:49.267954    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:00:53.410338 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:53.410314    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"]
Sep 17 21:00:53.412341 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:00:53.412306    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda444ff21_4251_4065_b64d_cf4b89173089.slice/crio-d105048457f8024300c2129b62a7792c1b0637f87f74c358090bc31e4bf53896 WatchSource:0}: Error finding container d105048457f8024300c2129b62a7792c1b0637f87f74c358090bc31e4bf53896: Status 404 returned error can't find the container with id d105048457f8024300c2129b62a7792c1b0637f87f74c358090bc31e4bf53896
Sep 17 21:00:54.017130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:54.017084    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerStarted","Data":"ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff"}
Sep 17 21:00:54.017299 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:54.017158    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerStarted","Data":"d105048457f8024300c2129b62a7792c1b0637f87f74c358090bc31e4bf53896"}
Sep 17 21:00:54.019446 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:54.019418    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerStarted","Data":"6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f"}
Sep 17 21:00:57.035998 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:57.035964    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerStarted","Data":"7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711"}
Sep 17 21:00:58.043304 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:58.043249    2586 generic.go:356] "Generic (PLEG): container finished" podID="a444ff21-4251-4065-b64d-cf4b89173089" containerID="ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff" exitCode=0
Sep 17 21:00:58.043752 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:58.043297    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerDied","Data":"ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff"}
Sep 17 21:00:59.049347 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:59.049314    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerStarted","Data":"0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a"}
Sep 17 21:00:59.049681 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:00:59.049492    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:01:00.057375 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:00.057229    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:01:00.057375 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:00.057306    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:01:00.059952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:00.059247    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:00.060395 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:00.060312    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:01.064713 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:01.063749    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:01.064713 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:01.064589    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:01.071569 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:01.071369    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:01:01.092209 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:01.092156    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podStartSLOduration=4.329892244 podStartE2EDuration="28.092140827s" totalImagesPullingTime="19.909122529s" totalInitContainerRuntime="3.853126054s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:00:59.086736145 +0000 UTC m=+674.620675955" watchObservedRunningTime="2026-09-17 21:01:01.092140827 +0000 UTC m=+676.626080637"
Sep 17 21:01:02.066988 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:02.066946    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:02.067469 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:02.067311    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:12.067821 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:12.067771    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:12.068350 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:12.068276    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:22.067813 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:22.067770    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:22.068281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:22.068245    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:23.159140 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:23.159108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerStarted","Data":"c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec"}
Sep 17 21:01:23.159521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:23.159168    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerStarted","Data":"cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480"}
Sep 17 21:01:23.159521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:23.159385    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:01:23.178547 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:23.178504    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podStartSLOduration=6.543513829 podStartE2EDuration="35.178490082s" totalImagesPullingTime="24.630814431s" totalInitContainerRuntime="4.004161822s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:01:23.176892046 +0000 UTC m=+698.710831855" watchObservedRunningTime="2026-09-17 21:01:23.178490082 +0000 UTC m=+698.712429891"
Sep 17 21:01:24.163185 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:24.163154    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:01:24.164439 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:24.164411    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:01:25.166266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:25.166230    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:01:30.170586 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:30.170554    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:01:30.171106 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:30.171038    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:01:32.067274 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:32.067223    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:32.067697 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:32.067608    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:40.170956 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:40.170912    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:01:42.066905 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:42.066854    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:42.067300 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:42.067250    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:01:50.171009 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:50.170970    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:01:52.067743 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:52.067696    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:01:52.068189 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:01:52.068164    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:00.171311 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:00.171274    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:02:02.067611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:02.067556    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:02:02.068003 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:02.067953    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:10.171025 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:10.170978    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:02:12.066912 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:12.066863    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:02:12.067308 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:12.067277    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:19.667045 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:19.667015    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:02:19.667509 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:19.667306    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:02:20.171150 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:20.171115    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:02:29.328738 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.328705    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"]
Sep 17 21:02:29.329284 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.329014    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" containerID="cri-o://6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f" gracePeriod=30
Sep 17 21:02:29.329284 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.329027    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" containerID="cri-o://0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a" gracePeriod=30
Sep 17 21:02:29.329284 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.329044    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy" containerID="cri-o://7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711" gracePeriod=30
Sep 17 21:02:29.504011 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.503985    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"]
Sep 17 21:02:29.506100 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.506084    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.512761 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.511117    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 21:02:29.512761 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.511123    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 21:02:29.521963 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.521942    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"]
Sep 17 21:02:29.565916 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.565888    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.566036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.565922    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.566036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.566015    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzlr\" (UniqueName: \"kubernetes.io/projected/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-api-access-qqzlr\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.566119 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.566048    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d18304-38f8-44cf-bf9d-cee1c4687a5c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.566119 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.566066    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.666756 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.666720    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzlr\" (UniqueName: \"kubernetes.io/projected/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-api-access-qqzlr\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.666897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.666763    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d18304-38f8-44cf-bf9d-cee1c4687a5c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.666897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.666788    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.666897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.666751    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:02:29.667090 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.667064    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:29.667203 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.667176    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.667203 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.667187    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.667332 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.667219    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.667698 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.667682    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.667770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.667754    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.669166 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.669147    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d18304-38f8-44cf-bf9d-cee1c4687a5c-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.675370 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.675339    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzlr\" (UniqueName: \"kubernetes.io/projected/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-api-access-qqzlr\") pod \"isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:29.818047 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:29.818011    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:30.094330 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:30.094256    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"]
Sep 17 21:02:30.097676 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:02:30.097649    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d18304_38f8_44cf_bf9d_cee1c4687a5c.slice/crio-67eef0f737d7596b26bd2d2c15de5a2df3bfb8b51ce9fd5602f7ae941243b297 WatchSource:0}: Error finding container 67eef0f737d7596b26bd2d2c15de5a2df3bfb8b51ce9fd5602f7ae941243b297: Status 404 returned error can't find the container with id 67eef0f737d7596b26bd2d2c15de5a2df3bfb8b51ce9fd5602f7ae941243b297
Sep 17 21:02:30.171212 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:30.171183    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:02:30.411473 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:30.411437    2586 generic.go:356] "Generic (PLEG): container finished" podID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerID="7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711" exitCode=2
Sep 17 21:02:30.411881 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:30.411517    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerDied","Data":"7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711"}
Sep 17 21:02:30.413496 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:30.413467    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerStarted","Data":"5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da"}
Sep 17 21:02:30.413621 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:30.413539    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerStarted","Data":"67eef0f737d7596b26bd2d2c15de5a2df3bfb8b51ce9fd5602f7ae941243b297"}
Sep 17 21:02:31.062845 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:31.062804    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 21:02:34.432312 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:34.432240    2586 generic.go:356] "Generic (PLEG): container finished" podID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerID="6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f" exitCode=0
Sep 17 21:02:34.432701 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:34.432316    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerDied","Data":"6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f"}
Sep 17 21:02:34.433867 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:34.433845    2586 generic.go:356] "Generic (PLEG): container finished" podID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerID="5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da" exitCode=0
Sep 17 21:02:34.433982 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:34.433908    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerDied","Data":"5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da"}
Sep 17 21:02:35.439816 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:35.439784    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerStarted","Data":"5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4"}
Sep 17 21:02:35.440266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:35.439859    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerStarted","Data":"f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32"}
Sep 17 21:02:35.440266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:35.439898    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerStarted","Data":"66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9"}
Sep 17 21:02:35.440266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:35.440139    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:35.463936 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:35.463895    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podStartSLOduration=2.482983391 podStartE2EDuration="6.463880215s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.980896824s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:02:35.462181947 +0000 UTC m=+770.996121754" watchObservedRunningTime="2026-09-17 21:02:35.463880215 +0000 UTC m=+770.997820024"
Sep 17 21:02:36.062858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:36.062819    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 21:02:36.443879 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:36.443842    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:36.444268 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:36.443931    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:36.445213 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:36.445179    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:02:36.445860 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:36.445836    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:37.447730 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:37.447679    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:02:37.448156 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:37.448134    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:39.666977 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:39.666925    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:02:39.667412 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:39.667240    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:40.171531 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:40.171503    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:02:41.062728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:41.062665    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 21:02:41.063123 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:41.062861    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:02:42.453251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:42.453224    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:02:42.453827 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:42.453799    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:02:42.454054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:42.454034    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:46.063515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:46.063477    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 21:02:49.430224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:49.430195    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"]
Sep 17 21:02:49.430645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:49.430426    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" containerID="cri-o://cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480" gracePeriod=30
Sep 17 21:02:49.430645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:49.430482    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kube-rbac-proxy" containerID="cri-o://c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec" gracePeriod=30
Sep 17 21:02:49.667277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:49.667231    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 21:02:49.667451 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:49.667414    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:02:49.667642 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:49.667614    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:49.667748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:49.667735    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:02:50.167275 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:50.167233    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 21:02:50.171580 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:50.171551    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 21:02:50.501300 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:50.501216    2586 generic.go:356] "Generic (PLEG): container finished" podID="a444ff21-4251-4065-b64d-cf4b89173089" containerID="c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec" exitCode=2
Sep 17 21:02:50.501300 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:50.501290    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerDied","Data":"c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec"}
Sep 17 21:02:51.063520 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:51.063478    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 21:02:52.453770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:52.453730    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:02:52.454210 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:52.454122    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:02:54.357506 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.357484    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:02:54.449076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449054    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-kube-rbac-proxy-sar-config\") pod \"a444ff21-4251-4065-b64d-cf4b89173089\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") "
Sep 17 21:02:54.449188 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449088    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-cabundle-cert\") pod \"a444ff21-4251-4065-b64d-cf4b89173089\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") "
Sep 17 21:02:54.449188 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449135    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg2kc\" (UniqueName: \"kubernetes.io/projected/a444ff21-4251-4065-b64d-cf4b89173089-kube-api-access-vg2kc\") pod \"a444ff21-4251-4065-b64d-cf4b89173089\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") "
Sep 17 21:02:54.449293 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449269    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a444ff21-4251-4065-b64d-cf4b89173089-proxy-tls\") pod \"a444ff21-4251-4065-b64d-cf4b89173089\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") "
Sep 17 21:02:54.449347 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449309    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a444ff21-4251-4065-b64d-cf4b89173089-kserve-provision-location\") pod \"a444ff21-4251-4065-b64d-cf4b89173089\" (UID: \"a444ff21-4251-4065-b64d-cf4b89173089\") "
Sep 17 21:02:54.449447 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449415    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a444ff21-4251-4065-b64d-cf4b89173089" (UID: "a444ff21-4251-4065-b64d-cf4b89173089"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:02:54.449505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449469    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a444ff21-4251-4065-b64d-cf4b89173089" (UID: "a444ff21-4251-4065-b64d-cf4b89173089"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:02:54.449624 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449604    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:54.449723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449627    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a444ff21-4251-4065-b64d-cf4b89173089-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:54.449723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.449632    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a444ff21-4251-4065-b64d-cf4b89173089-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a444ff21-4251-4065-b64d-cf4b89173089" (UID: "a444ff21-4251-4065-b64d-cf4b89173089"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:02:54.451271 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.451253    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a444ff21-4251-4065-b64d-cf4b89173089-kube-api-access-vg2kc" (OuterVolumeSpecName: "kube-api-access-vg2kc") pod "a444ff21-4251-4065-b64d-cf4b89173089" (UID: "a444ff21-4251-4065-b64d-cf4b89173089"). InnerVolumeSpecName "kube-api-access-vg2kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:02:54.451388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.451349    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a444ff21-4251-4065-b64d-cf4b89173089-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a444ff21-4251-4065-b64d-cf4b89173089" (UID: "a444ff21-4251-4065-b64d-cf4b89173089"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:02:54.519073 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.519015    2586 generic.go:356] "Generic (PLEG): container finished" podID="a444ff21-4251-4065-b64d-cf4b89173089" containerID="cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480" exitCode=0
Sep 17 21:02:54.519175 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.519070    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"
Sep 17 21:02:54.519175 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.519091    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerDied","Data":"cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480"}
Sep 17 21:02:54.519175 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.519163    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k" event={"ID":"a444ff21-4251-4065-b64d-cf4b89173089","Type":"ContainerDied","Data":"d105048457f8024300c2129b62a7792c1b0637f87f74c358090bc31e4bf53896"}
Sep 17 21:02:54.519286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.519194    2586 scope.go:117] "RemoveContainer" containerID="c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec"
Sep 17 21:02:54.527941 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.527925    2586 scope.go:117] "RemoveContainer" containerID="cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480"
Sep 17 21:02:54.537038 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.537017    2586 scope.go:117] "RemoveContainer" containerID="ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff"
Sep 17 21:02:54.541646 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.541616    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"]
Sep 17 21:02:54.545481 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.545430    2586 scope.go:117] "RemoveContainer" containerID="c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec"
Sep 17 21:02:54.546246 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:54.546112    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec\": container with ID starting with c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec not found: ID does not exist" containerID="c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec"
Sep 17 21:02:54.546246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.546147    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec"} err="failed to get container status \"c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec\": rpc error: code = NotFound desc = could not find container \"c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec\": container with ID starting with c6ea96a2e565e9f404cde7f4d25f3734836d2bfe8cd80ddfd98052c764a29aec not found: ID does not exist"
Sep 17 21:02:54.546246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.546172    2586 scope.go:117] "RemoveContainer" containerID="cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480"
Sep 17 21:02:54.546522 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:54.546495    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480\": container with ID starting with cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480 not found: ID does not exist" containerID="cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480"
Sep 17 21:02:54.546582 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.546531    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480"} err="failed to get container status \"cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480\": rpc error: code = NotFound desc = could not find container \"cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480\": container with ID starting with cb1324560e8395eb94f5623febe5673fe93b0dd2ac9ae64b6e1b8d05dc99a480 not found: ID does not exist"
Sep 17 21:02:54.546582 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.546553    2586 scope.go:117] "RemoveContainer" containerID="ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff"
Sep 17 21:02:54.546835 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:54.546810    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff\": container with ID starting with ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff not found: ID does not exist" containerID="ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff"
Sep 17 21:02:54.546943 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.546840    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff"} err="failed to get container status \"ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff\": rpc error: code = NotFound desc = could not find container \"ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff\": container with ID starting with ed33f3cc2d1e3c4aba358c44f9d9d02cb42464abb7a27d787f024160eeadc1ff not found: ID does not exist"
Sep 17 21:02:54.547695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.547676    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-8jg8k"]
Sep 17 21:02:54.550317 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.550299    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg2kc\" (UniqueName: \"kubernetes.io/projected/a444ff21-4251-4065-b64d-cf4b89173089-kube-api-access-vg2kc\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:54.550317 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.550318    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a444ff21-4251-4065-b64d-cf4b89173089-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:54.550531 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.550328    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a444ff21-4251-4065-b64d-cf4b89173089-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:54.656947 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.656912    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"]
Sep 17 21:02:54.657204 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657191    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kube-rbac-proxy"
Sep 17 21:02:54.657252 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657207    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kube-rbac-proxy"
Sep 17 21:02:54.657252 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657221    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container"
Sep 17 21:02:54.657252 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657226    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container"
Sep 17 21:02:54.657252 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657236    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="storage-initializer"
Sep 17 21:02:54.657252 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657243    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="storage-initializer"
Sep 17 21:02:54.657409 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657287    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kube-rbac-proxy"
Sep 17 21:02:54.657409 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.657295    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a444ff21-4251-4065-b64d-cf4b89173089" containerName="kserve-container"
Sep 17 21:02:54.660177 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.660160    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.662861 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.662838    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 21:02:54.663908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.663890    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 21:02:54.670676 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.670654    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a444ff21-4251-4065-b64d-cf4b89173089" path="/var/lib/kubelet/pods/a444ff21-4251-4065-b64d-cf4b89173089/volumes"
Sep 17 21:02:54.684415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.684392    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"]
Sep 17 21:02:54.752308 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.752282    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.752430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.752319    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd29w\" (UniqueName: \"kubernetes.io/projected/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-api-access-qd29w\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.752430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.752349    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.752430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.752420    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.752531 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.752454    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.853816 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.853734    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.853816 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.853781    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.853816 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.853810    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd29w\" (UniqueName: \"kubernetes.io/projected/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-api-access-qd29w\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.854025 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.853831    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.854025 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.853858    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.854128 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:54.854038    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 21:02:54.854128 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:54.854110    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls podName:ddd1feaa-77a7-4a54-9ed8-9e5861055643 nodeName:}" failed. No retries permitted until 2026-09-17 21:02:55.354089045 +0000 UTC m=+790.888028845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-65m8p" (UID: "ddd1feaa-77a7-4a54-9ed8-9e5861055643") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Sep 17 21:02:54.854231 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.854197    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.854481 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.854464    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.854565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.854547    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:54.862651 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:54.862633    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd29w\" (UniqueName: \"kubernetes.io/projected/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-api-access-qd29w\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:55.358537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:55.358503    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:55.360831 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:55.360808    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-65m8p\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:55.569950 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:55.569920    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:02:55.850146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:55.850113    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"]
Sep 17 21:02:55.853049 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:02:55.853023    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd1feaa_77a7_4a54_9ed8_9e5861055643.slice/crio-863c9e46145d8e6149ec575aaf62f5ef8565eb9965473370700951f4409532f4 WatchSource:0}: Error finding container 863c9e46145d8e6149ec575aaf62f5ef8565eb9965473370700951f4409532f4: Status 404 returned error can't find the container with id 863c9e46145d8e6149ec575aaf62f5ef8565eb9965473370700951f4409532f4
Sep 17 21:02:56.063087 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:56.063050    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 21:02:56.529082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:56.529043    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerStarted","Data":"063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c"}
Sep 17 21:02:56.529502 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:56.529108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerStarted","Data":"863c9e46145d8e6149ec575aaf62f5ef8565eb9965473370700951f4409532f4"}
Sep 17 21:02:59.470078 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.470055    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:02:59.543190 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.543148    2586 generic.go:356] "Generic (PLEG): container finished" podID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerID="0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a" exitCode=0
Sep 17 21:02:59.543388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.543207    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerDied","Data":"0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a"}
Sep 17 21:02:59.543388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.543216    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"
Sep 17 21:02:59.543388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.543277    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72" event={"ID":"272ec759-4267-4ae4-aeb0-cd3ac5382a64","Type":"ContainerDied","Data":"8c24d54cb934d97714f37b7fe53052fdc90a3f3511ba53e2258fd024b124c826"}
Sep 17 21:02:59.543388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.543319    2586 scope.go:117] "RemoveContainer" containerID="0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a"
Sep 17 21:02:59.552420 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.552400    2586 scope.go:117] "RemoveContainer" containerID="7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711"
Sep 17 21:02:59.560672 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.560656    2586 scope.go:117] "RemoveContainer" containerID="6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f"
Sep 17 21:02:59.589848 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.589800    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-cabundle-cert\") pod \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") "
Sep 17 21:02:59.589848 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.589832    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndhxw\" (UniqueName: \"kubernetes.io/projected/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-api-access-ndhxw\") pod \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") "
Sep 17 21:02:59.589978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.589866    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kserve-provision-location\") pod \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") "
Sep 17 21:02:59.589978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.589904    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272ec759-4267-4ae4-aeb0-cd3ac5382a64-proxy-tls\") pod \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") "
Sep 17 21:02:59.589978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.589929    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-rbac-proxy-sar-config\") pod \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\" (UID: \"272ec759-4267-4ae4-aeb0-cd3ac5382a64\") "
Sep 17 21:02:59.590232 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.590207    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "272ec759-4267-4ae4-aeb0-cd3ac5382a64" (UID: "272ec759-4267-4ae4-aeb0-cd3ac5382a64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:02:59.590269 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.590233    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "272ec759-4267-4ae4-aeb0-cd3ac5382a64" (UID: "272ec759-4267-4ae4-aeb0-cd3ac5382a64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:02:59.590311 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.590294    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "272ec759-4267-4ae4-aeb0-cd3ac5382a64" (UID: "272ec759-4267-4ae4-aeb0-cd3ac5382a64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:02:59.591689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.591671    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/272ec759-4267-4ae4-aeb0-cd3ac5382a64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "272ec759-4267-4ae4-aeb0-cd3ac5382a64" (UID: "272ec759-4267-4ae4-aeb0-cd3ac5382a64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:02:59.591813 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.591795    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-api-access-ndhxw" (OuterVolumeSpecName: "kube-api-access-ndhxw") pod "272ec759-4267-4ae4-aeb0-cd3ac5382a64" (UID: "272ec759-4267-4ae4-aeb0-cd3ac5382a64"). InnerVolumeSpecName "kube-api-access-ndhxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:02:59.636296 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.636278    2586 scope.go:117] "RemoveContainer" containerID="aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6"
Sep 17 21:02:59.643232 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.643215    2586 scope.go:117] "RemoveContainer" containerID="0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a"
Sep 17 21:02:59.643506 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:59.643485    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a\": container with ID starting with 0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a not found: ID does not exist" containerID="0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a"
Sep 17 21:02:59.643549 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.643516    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a"} err="failed to get container status \"0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a\": rpc error: code = NotFound desc = could not find container \"0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a\": container with ID starting with 0ad98cd097d97b4d8233d2fb1c2e34aefb7185d337561be635d8fea3a34a151a not found: ID does not exist"
Sep 17 21:02:59.643549 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.643533    2586 scope.go:117] "RemoveContainer" containerID="7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711"
Sep 17 21:02:59.643773 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:59.643753    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711\": container with ID starting with 7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711 not found: ID does not exist" containerID="7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711"
Sep 17 21:02:59.643842 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.643791    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711"} err="failed to get container status \"7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711\": rpc error: code = NotFound desc = could not find container \"7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711\": container with ID starting with 7ba0a01fb2c758cfbe808a2c6e46d283dce6b16375d5c4a0f868af4bfe185711 not found: ID does not exist"
Sep 17 21:02:59.643842 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.643814    2586 scope.go:117] "RemoveContainer" containerID="6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f"
Sep 17 21:02:59.644038 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:59.644023    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f\": container with ID starting with 6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f not found: ID does not exist" containerID="6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f"
Sep 17 21:02:59.644080 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.644042    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f"} err="failed to get container status \"6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f\": rpc error: code = NotFound desc = could not find container \"6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f\": container with ID starting with 6ece8fc74a434081732b1530cce09a8847fd2e9ad0dd84b769d001d75c7d656f not found: ID does not exist"
Sep 17 21:02:59.644080 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.644055    2586 scope.go:117] "RemoveContainer" containerID="aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6"
Sep 17 21:02:59.644275 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:02:59.644260    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6\": container with ID starting with aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6 not found: ID does not exist" containerID="aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6"
Sep 17 21:02:59.644321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.644281    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6"} err="failed to get container status \"aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6\": rpc error: code = NotFound desc = could not find container \"aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6\": container with ID starting with aeb063b6dde4e827e4c10d373083b306eb96b4c59059d6b300f4ed9df5b439f6 not found: ID does not exist"
Sep 17 21:02:59.690849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.690827    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:59.690849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.690848    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndhxw\" (UniqueName: \"kubernetes.io/projected/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-api-access-ndhxw\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:59.690952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.690857    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:59.690952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.690866    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/272ec759-4267-4ae4-aeb0-cd3ac5382a64-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:59.690952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.690874    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/272ec759-4267-4ae4-aeb0-cd3ac5382a64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:02:59.866942 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.866904    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"]
Sep 17 21:02:59.868312 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:02:59.868292    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77774898d8-sct72"]
Sep 17 21:03:00.548830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:00.548799    2586 generic.go:356] "Generic (PLEG): container finished" podID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerID="063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c" exitCode=0
Sep 17 21:03:00.549335 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:00.548879    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerDied","Data":"063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c"}
Sep 17 21:03:00.671083 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:00.671061    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" path="/var/lib/kubelet/pods/272ec759-4267-4ae4-aeb0-cd3ac5382a64/volumes"
Sep 17 21:03:01.555326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:01.555291    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerStarted","Data":"e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2"}
Sep 17 21:03:01.555760 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:01.555406    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerStarted","Data":"9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941"}
Sep 17 21:03:01.555760 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:01.555608    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:03:01.555760 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:01.555663    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:03:01.556794 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:01.556769    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:03:01.573782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:01.573732    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podStartSLOduration=3.92043163 podStartE2EDuration="7.573681375s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653249745s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:03:01.573305934 +0000 UTC m=+797.107245744" watchObservedRunningTime="2026-09-17 21:03:01.573681375 +0000 UTC m=+797.107621188"
Sep 17 21:03:02.453822 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:02.453786    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:03:02.454266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:02.454243    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:02.560200 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:02.560162    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:03:07.564748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:07.564721    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:03:07.565195 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:07.565171    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:03:12.454200 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:12.454157    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:03:12.454612 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:12.454528    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:17.566067 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:17.566029    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:03:22.453814 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:22.453773    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:03:22.454222 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:22.454036    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:27.566074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:27.566030    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:03:32.454537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:32.454496    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:03:32.455007 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:32.454983    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:37.565923 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:37.565886    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:03:42.454241 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:42.454194    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:03:42.454684 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:42.454657    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:03:47.565147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:47.565109    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:03:52.454567 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:52.454535    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:03:52.455019 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:52.454622    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:03:57.565577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:03:57.565538    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:04:04.963835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:04.963799    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"]
Sep 17 21:04:04.964448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:04.964073    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" containerID="cri-o://66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9" gracePeriod=30
Sep 17 21:04:04.964448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:04.964112    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" containerID="cri-o://5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4" gracePeriod=30
Sep 17 21:04:04.964448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:04.964122    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy" containerID="cri-o://f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32" gracePeriod=30
Sep 17 21:04:05.110592 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110560    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"]
Sep 17 21:04:05.110855 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110842    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy"
Sep 17 21:04:05.110898 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110859    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy"
Sep 17 21:04:05.110898 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110883    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent"
Sep 17 21:04:05.110898 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110891    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent"
Sep 17 21:04:05.111005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110901    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="storage-initializer"
Sep 17 21:04:05.111005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110907    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="storage-initializer"
Sep 17 21:04:05.111005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110919    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container"
Sep 17 21:04:05.111005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110924    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container"
Sep 17 21:04:05.111005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110969    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="agent"
Sep 17 21:04:05.111005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110980    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kube-rbac-proxy"
Sep 17 21:04:05.111005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.110989    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="272ec759-4267-4ae4-aeb0-cd3ac5382a64" containerName="kserve-container"
Sep 17 21:04:05.114024 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.114008    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.116728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.116712    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 21:04:05.116846 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.116824    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 21:04:05.125273 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.125253    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"]
Sep 17 21:04:05.165830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.165803    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c7695e-b061-4d60-8187-4198cff4441e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.165936 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.165834    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c7695e-b061-4d60-8187-4198cff4441e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.165936 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.165858    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gx7d\" (UniqueName: \"kubernetes.io/projected/19c7695e-b061-4d60-8187-4198cff4441e-kube-api-access-7gx7d\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.266332 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.266265    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gx7d\" (UniqueName: \"kubernetes.io/projected/19c7695e-b061-4d60-8187-4198cff4441e-kube-api-access-7gx7d\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.266332 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.266322    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c7695e-b061-4d60-8187-4198cff4441e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.266543 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.266344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c7695e-b061-4d60-8187-4198cff4441e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.266954 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.266926    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c7695e-b061-4d60-8187-4198cff4441e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.268832 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.268808    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c7695e-b061-4d60-8187-4198cff4441e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.274054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.274033    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gx7d\" (UniqueName: \"kubernetes.io/projected/19c7695e-b061-4d60-8187-4198cff4441e-kube-api-access-7gx7d\") pod \"message-dumper-predictor-6575bcfdfd-9whlx\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.423473 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.423440    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:05.691833 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.691799    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"]
Sep 17 21:04:05.694910 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:04:05.694884    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c7695e_b061_4d60_8187_4198cff4441e.slice/crio-6a8b5228f6b31f695895623c2dfd4cafb828b98fa96d4f660b056120c90bad61 WatchSource:0}: Error finding container 6a8b5228f6b31f695895623c2dfd4cafb828b98fa96d4f660b056120c90bad61: Status 404 returned error can't find the container with id 6a8b5228f6b31f695895623c2dfd4cafb828b98fa96d4f660b056120c90bad61
Sep 17 21:04:05.696801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.696786    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:04:05.804109 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.804074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" event={"ID":"19c7695e-b061-4d60-8187-4198cff4441e","Type":"ContainerStarted","Data":"6a8b5228f6b31f695895623c2dfd4cafb828b98fa96d4f660b056120c90bad61"}
Sep 17 21:04:05.806483 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.806461    2586 generic.go:356] "Generic (PLEG): container finished" podID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerID="f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32" exitCode=2
Sep 17 21:04:05.806591 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:05.806510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerDied","Data":"f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32"}
Sep 17 21:04:07.448283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:07.448243    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 21:04:07.565745 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:07.565713    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:04:07.817036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:07.816953    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" event={"ID":"19c7695e-b061-4d60-8187-4198cff4441e","Type":"ContainerStarted","Data":"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"}
Sep 17 21:04:07.817036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:07.817007    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" event={"ID":"19c7695e-b061-4d60-8187-4198cff4441e","Type":"ContainerStarted","Data":"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"}
Sep 17 21:04:07.817217 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:07.817084    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:07.843816 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:07.843767    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" podStartSLOduration=1.769144115 podStartE2EDuration="2.843751766s" totalImagesPullingTime="1.074607651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:07.843543971 +0000 UTC m=+863.377483774" watchObservedRunningTime="2026-09-17 21:04:07.843751766 +0000 UTC m=+863.377691574"
Sep 17 21:04:08.821046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:08.821011    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:08.822707 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:08.822683    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:09.826979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:09.826894    2586 generic.go:356] "Generic (PLEG): container finished" podID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerID="66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9" exitCode=0
Sep 17 21:04:09.826979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:09.826965    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerDied","Data":"66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9"}
Sep 17 21:04:12.448530 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:12.448491    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 21:04:12.453829 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:12.453802    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:04:12.454216 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:12.454188    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:04:15.835971 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:15.835943    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:04:17.448323 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:17.448289    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 21:04:17.448701 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:17.448428    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:04:17.566441 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:17.566404    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:04:22.448758 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:22.448709    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 21:04:22.454105 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:22.454074    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:04:22.454379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:22.454337    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:04:25.119879 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.119846    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"]
Sep 17 21:04:25.122939 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.122915    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.125275 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.125255    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 21:04:25.125275 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.125267    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 21:04:25.139594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.139563    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"]
Sep 17 21:04:25.197334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.197305    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"]
Sep 17 21:04:25.197666 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.197624    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" containerID="cri-o://9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941" gracePeriod=30
Sep 17 21:04:25.197781 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.197640    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kube-rbac-proxy" containerID="cri-o://e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2" gracePeriod=30
Sep 17 21:04:25.204644 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.204623    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-cabundle-cert\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.204747 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.204661    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d27677-8197-461d-891e-73308d02f186-kserve-provision-location\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.204747 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.204687    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d27677-8197-461d-891e-73308d02f186-proxy-tls\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.204840 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.204770    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.204874 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.204833    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85mm6\" (UniqueName: \"kubernetes.io/projected/d3d27677-8197-461d-891e-73308d02f186-kube-api-access-85mm6\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.305607 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.305575    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-cabundle-cert\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.305766 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.305618    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d27677-8197-461d-891e-73308d02f186-kserve-provision-location\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.305766 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.305650    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d27677-8197-461d-891e-73308d02f186-proxy-tls\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.305766 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.305672    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.305766 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.305695    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85mm6\" (UniqueName: \"kubernetes.io/projected/d3d27677-8197-461d-891e-73308d02f186-kube-api-access-85mm6\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.306077 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.306049    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d27677-8197-461d-891e-73308d02f186-kserve-provision-location\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.306439 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.306417    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-cabundle-cert\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.306518 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.306475    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.308142 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.308119    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d27677-8197-461d-891e-73308d02f186-proxy-tls\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.314444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.314421    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85mm6\" (UniqueName: \"kubernetes.io/projected/d3d27677-8197-461d-891e-73308d02f186-kube-api-access-85mm6\") pod \"isvc-logger-predictor-8559c4556c-drp48\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.456332 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.456249    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:25.729163 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.729138    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"]
Sep 17 21:04:25.731325 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:04:25.731301    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d27677_8197_461d_891e_73308d02f186.slice/crio-9b0d944da44a0a77773538e04a0130dcdab45067101161fd00d40b86db360e12 WatchSource:0}: Error finding container 9b0d944da44a0a77773538e04a0130dcdab45067101161fd00d40b86db360e12: Status 404 returned error can't find the container with id 9b0d944da44a0a77773538e04a0130dcdab45067101161fd00d40b86db360e12
Sep 17 21:04:25.889228 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.889195    2586 generic.go:356] "Generic (PLEG): container finished" podID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerID="e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2" exitCode=2
Sep 17 21:04:25.889390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.889266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerDied","Data":"e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2"}
Sep 17 21:04:25.890883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.890861    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerStarted","Data":"b5c01a77f0e63896216fb295c70f3cde95a3d970086e9177881688081c61051a"}
Sep 17 21:04:25.890973 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:25.890911    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerStarted","Data":"9b0d944da44a0a77773538e04a0130dcdab45067101161fd00d40b86db360e12"}
Sep 17 21:04:27.448558 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:27.448472    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 21:04:27.561173 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:27.561139    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 21:04:27.565476 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:27.565455    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 21:04:29.837438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.837387    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:04:29.906835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.906807    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3d27677-8197-461d-891e-73308d02f186" containerID="b5c01a77f0e63896216fb295c70f3cde95a3d970086e9177881688081c61051a" exitCode=0
Sep 17 21:04:29.906972 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.906879    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerDied","Data":"b5c01a77f0e63896216fb295c70f3cde95a3d970086e9177881688081c61051a"}
Sep 17 21:04:29.908916 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.908885    2586 generic.go:356] "Generic (PLEG): container finished" podID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerID="9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941" exitCode=0
Sep 17 21:04:29.909013 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.908932    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"
Sep 17 21:04:29.909013 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.908965    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerDied","Data":"9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941"}
Sep 17 21:04:29.909082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.909012    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p" event={"ID":"ddd1feaa-77a7-4a54-9ed8-9e5861055643","Type":"ContainerDied","Data":"863c9e46145d8e6149ec575aaf62f5ef8565eb9965473370700951f4409532f4"}
Sep 17 21:04:29.909082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.909039    2586 scope.go:117] "RemoveContainer" containerID="e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2"
Sep 17 21:04:29.919076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.919052    2586 scope.go:117] "RemoveContainer" containerID="9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941"
Sep 17 21:04:29.929938 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.929924    2586 scope.go:117] "RemoveContainer" containerID="063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c"
Sep 17 21:04:29.938061 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938040    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-cabundle-cert\") pod \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") "
Sep 17 21:04:29.938159 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938084    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kserve-provision-location\") pod \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") "
Sep 17 21:04:29.938159 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938123    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-rbac-proxy-sar-config\") pod \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") "
Sep 17 21:04:29.938267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938161    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls\") pod \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") "
Sep 17 21:04:29.938267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938208    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd29w\" (UniqueName: \"kubernetes.io/projected/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-api-access-qd29w\") pod \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\" (UID: \"ddd1feaa-77a7-4a54-9ed8-9e5861055643\") "
Sep 17 21:04:29.938396 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938368    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ddd1feaa-77a7-4a54-9ed8-9e5861055643" (UID: "ddd1feaa-77a7-4a54-9ed8-9e5861055643"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:04:29.938459 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938401    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ddd1feaa-77a7-4a54-9ed8-9e5861055643" (UID: "ddd1feaa-77a7-4a54-9ed8-9e5861055643"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:29.938584 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.938546    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ddd1feaa-77a7-4a54-9ed8-9e5861055643" (UID: "ddd1feaa-77a7-4a54-9ed8-9e5861055643"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:04:29.940526 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.940503    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-api-access-qd29w" (OuterVolumeSpecName: "kube-api-access-qd29w") pod "ddd1feaa-77a7-4a54-9ed8-9e5861055643" (UID: "ddd1feaa-77a7-4a54-9ed8-9e5861055643"). InnerVolumeSpecName "kube-api-access-qd29w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:04:29.940627 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.940581    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ddd1feaa-77a7-4a54-9ed8-9e5861055643" (UID: "ddd1feaa-77a7-4a54-9ed8-9e5861055643"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:04:29.941530 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.941507    2586 scope.go:117] "RemoveContainer" containerID="e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2"
Sep 17 21:04:29.941793 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:04:29.941774    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2\": container with ID starting with e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2 not found: ID does not exist" containerID="e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2"
Sep 17 21:04:29.941863 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.941806    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2"} err="failed to get container status \"e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2\": rpc error: code = NotFound desc = could not find container \"e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2\": container with ID starting with e84ce87ad0feb32fe50bd90c4ca4366d8d596c0f9e6b60a811e75c26328a0aa2 not found: ID does not exist"
Sep 17 21:04:29.941863 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.941828    2586 scope.go:117] "RemoveContainer" containerID="9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941"
Sep 17 21:04:29.942093 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:04:29.942076    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941\": container with ID starting with 9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941 not found: ID does not exist" containerID="9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941"
Sep 17 21:04:29.942145 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.942100    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941"} err="failed to get container status \"9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941\": rpc error: code = NotFound desc = could not find container \"9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941\": container with ID starting with 9f394314f9ae25103ba157d7d3f65fe5521ee011c66de0fabaad6b2433db3941 not found: ID does not exist"
Sep 17 21:04:29.942145 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.942115    2586 scope.go:117] "RemoveContainer" containerID="063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c"
Sep 17 21:04:29.942326 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:04:29.942307    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c\": container with ID starting with 063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c not found: ID does not exist" containerID="063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c"
Sep 17 21:04:29.942401 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:29.942328    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c"} err="failed to get container status \"063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c\": rpc error: code = NotFound desc = could not find container \"063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c\": container with ID starting with 063bfbc210e5353a87fdc7c87a1a3d5979ed93f2b0227fd36a710b4360beb25c not found: ID does not exist"
Sep 17 21:04:30.039198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.039175    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd29w\" (UniqueName: \"kubernetes.io/projected/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-api-access-qd29w\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:30.039292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.039199    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:30.039292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.039212    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:30.039292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.039221    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ddd1feaa-77a7-4a54-9ed8-9e5861055643-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:30.039292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.039231    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ddd1feaa-77a7-4a54-9ed8-9e5861055643-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:30.241825 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.241799    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"]
Sep 17 21:04:30.247019 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.246992    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-65m8p"]
Sep 17 21:04:30.406633 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406605    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"]
Sep 17 21:04:30.406888 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406876    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container"
Sep 17 21:04:30.406931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406889    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container"
Sep 17 21:04:30.406931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406902    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="storage-initializer"
Sep 17 21:04:30.406931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406908    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="storage-initializer"
Sep 17 21:04:30.406931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406918    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kube-rbac-proxy"
Sep 17 21:04:30.406931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406924    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kube-rbac-proxy"
Sep 17 21:04:30.407081 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406977    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kube-rbac-proxy"
Sep 17 21:04:30.407081 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.406988    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" containerName="kserve-container"
Sep 17 21:04:30.409979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.409965    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.412393 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.412371    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 21:04:30.412511 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.412393    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 21:04:30.422001 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.421979    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"]
Sep 17 21:04:30.542661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.542576    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5c789ea-c59b-435f-acbb-02d693325eb7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.542661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.542625    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5c789ea-c59b-435f-acbb-02d693325eb7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.542856 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.542692    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.542856 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.542717    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.542856 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.542750    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jt5z\" (UniqueName: \"kubernetes.io/projected/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-api-access-5jt5z\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.647925 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.647887    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.648438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.648403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jt5z\" (UniqueName: \"kubernetes.io/projected/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-api-access-5jt5z\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.648562 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.648486    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5c789ea-c59b-435f-acbb-02d693325eb7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.648562 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.648526    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5c789ea-c59b-435f-acbb-02d693325eb7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.648674 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.648567    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.648894 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.648871    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5c789ea-c59b-435f-acbb-02d693325eb7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.648967 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.648949    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.649190 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.649171    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.650886 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.650859    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5c789ea-c59b-435f-acbb-02d693325eb7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.656303 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.656283    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jt5z\" (UniqueName: \"kubernetes.io/projected/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-api-access-5jt5z\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.671205 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.671170    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd1feaa-77a7-4a54-9ed8-9e5861055643" path="/var/lib/kubelet/pods/ddd1feaa-77a7-4a54-9ed8-9e5861055643/volumes"
Sep 17 21:04:30.719530 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.719508    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:30.916714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.916676    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerStarted","Data":"ba23ccfbc1a1a19c626749ed1ed89bbd8bb11e9d679a098369ac22908a2d0f61"}
Sep 17 21:04:30.917149 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.916766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerStarted","Data":"21e48d47396c3c1cb33e846dc724774ef82a05b56cc137d37b9fee3c965aa801"}
Sep 17 21:04:30.917149 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.916808    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerStarted","Data":"37526268fd9f1e30ecfc1a3b7f11f155ba76070a53b9ba8308c98692d2871da0"}
Sep 17 21:04:30.917276 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.917152    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:30.917343 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.917323    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:30.918449 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.918421    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:04:30.963101 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:30.963063    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podStartSLOduration=2.176688467 podStartE2EDuration="5.963049397s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.78636093s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:30.962293911 +0000 UTC m=+886.496233720" watchObservedRunningTime="2026-09-17 21:04:30.963049397 +0000 UTC m=+886.496989206"
Sep 17 21:04:31.003234 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:31.003215    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"]
Sep 17 21:04:31.005694 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:04:31.005672    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c789ea_c59b_435f_acbb_02d693325eb7.slice/crio-0d42d49363b62778c1f227b85c093a88828590e3b41a9f3d375542cd40e55e92 WatchSource:0}: Error finding container 0d42d49363b62778c1f227b85c093a88828590e3b41a9f3d375542cd40e55e92: Status 404 returned error can't find the container with id 0d42d49363b62778c1f227b85c093a88828590e3b41a9f3d375542cd40e55e92
Sep 17 21:04:31.923329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:31.923296    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerStarted","Data":"f0a4ede9c00b4f4c8ce459a5db3bc366b59fdf8afef5f1680cfc0d0508a07976"}
Sep 17 21:04:31.923706 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:31.923375    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerStarted","Data":"0d42d49363b62778c1f227b85c093a88828590e3b41a9f3d375542cd40e55e92"}
Sep 17 21:04:31.923706 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:31.923677    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:31.923835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:31.923808    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:04:31.924966 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:31.924940    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:04:32.448729 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:32.448682    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 21:04:32.454140 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:32.454114    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:5000: connect: connection refused"
Sep 17 21:04:32.454239 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:32.454225    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:04:32.454406 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:32.454384    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:04:32.454529 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:32.454484    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:04:32.927188 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:32.927152    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:04:32.927736 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:32.927712    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:04:34.935214 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:34.935185    2586 generic.go:356] "Generic (PLEG): container finished" podID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerID="f0a4ede9c00b4f4c8ce459a5db3bc366b59fdf8afef5f1680cfc0d0508a07976" exitCode=0
Sep 17 21:04:34.935548 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:34.935259    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerDied","Data":"f0a4ede9c00b4f4c8ce459a5db3bc366b59fdf8afef5f1680cfc0d0508a07976"}
Sep 17 21:04:35.602663 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.602636    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:04:35.689240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689210    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqzlr\" (UniqueName: \"kubernetes.io/projected/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-api-access-qqzlr\") pod \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") "
Sep 17 21:04:35.689437 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689250    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d18304-38f8-44cf-bf9d-cee1c4687a5c-proxy-tls\") pod \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") "
Sep 17 21:04:35.689437 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689304    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kserve-provision-location\") pod \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") "
Sep 17 21:04:35.689437 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689334    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-cabundle-cert\") pod \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") "
Sep 17 21:04:35.689437 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689404    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-rbac-proxy-sar-config\") pod \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\" (UID: \"44d18304-38f8-44cf-bf9d-cee1c4687a5c\") "
Sep 17 21:04:35.689685 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689646    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44d18304-38f8-44cf-bf9d-cee1c4687a5c" (UID: "44d18304-38f8-44cf-bf9d-cee1c4687a5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:04:35.689777 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689752    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44d18304-38f8-44cf-bf9d-cee1c4687a5c" (UID: "44d18304-38f8-44cf-bf9d-cee1c4687a5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:04:35.689820 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.689803    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44d18304-38f8-44cf-bf9d-cee1c4687a5c" (UID: "44d18304-38f8-44cf-bf9d-cee1c4687a5c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:04:35.691401 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.691378    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-api-access-qqzlr" (OuterVolumeSpecName: "kube-api-access-qqzlr") pod "44d18304-38f8-44cf-bf9d-cee1c4687a5c" (UID: "44d18304-38f8-44cf-bf9d-cee1c4687a5c"). InnerVolumeSpecName "kube-api-access-qqzlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:04:35.691451 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.691408    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d18304-38f8-44cf-bf9d-cee1c4687a5c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44d18304-38f8-44cf-bf9d-cee1c4687a5c" (UID: "44d18304-38f8-44cf-bf9d-cee1c4687a5c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:04:35.790658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.790595    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:35.790658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.790617    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:35.790658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.790626    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqzlr\" (UniqueName: \"kubernetes.io/projected/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kube-api-access-qqzlr\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:35.790658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.790635    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44d18304-38f8-44cf-bf9d-cee1c4687a5c-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:35.790658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.790645    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44d18304-38f8-44cf-bf9d-cee1c4687a5c-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:04:35.942459 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.942424    2586 generic.go:356] "Generic (PLEG): container finished" podID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerID="5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4" exitCode=0
Sep 17 21:04:35.942849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.942496    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"
Sep 17 21:04:35.942849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.942498    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerDied","Data":"5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4"}
Sep 17 21:04:35.942849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.942650    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt" event={"ID":"44d18304-38f8-44cf-bf9d-cee1c4687a5c","Type":"ContainerDied","Data":"67eef0f737d7596b26bd2d2c15de5a2df3bfb8b51ce9fd5602f7ae941243b297"}
Sep 17 21:04:35.942849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.942697    2586 scope.go:117] "RemoveContainer" containerID="5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4"
Sep 17 21:04:35.944806 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.944776    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerStarted","Data":"9a6ec940293822ff01c7a781593dad49b428b9ed873b7ca8efa3439f768d5fee"}
Sep 17 21:04:35.944908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.944848    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerStarted","Data":"dadf588d834a59be7d0e446635117732f97aef87521385630d00eef63496ba68"}
Sep 17 21:04:35.945071 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.945057    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:35.950763 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.950745    2586 scope.go:117] "RemoveContainer" containerID="f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32"
Sep 17 21:04:35.957788 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.957773    2586 scope.go:117] "RemoveContainer" containerID="66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9"
Sep 17 21:04:35.966507 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.966485    2586 scope.go:117] "RemoveContainer" containerID="5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da"
Sep 17 21:04:35.972229 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.972192    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podStartSLOduration=2.176480967 podStartE2EDuration="5.972180054s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.795699087s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:04:35.970755372 +0000 UTC m=+891.504695184" watchObservedRunningTime="2026-09-17 21:04:35.972180054 +0000 UTC m=+891.506119863"
Sep 17 21:04:35.974196 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.974170    2586 scope.go:117] "RemoveContainer" containerID="5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4"
Sep 17 21:04:35.974453 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:04:35.974434    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4\": container with ID starting with 5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4 not found: ID does not exist" containerID="5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4"
Sep 17 21:04:35.974589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.974459    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4"} err="failed to get container status \"5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4\": rpc error: code = NotFound desc = could not find container \"5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4\": container with ID starting with 5b5c2caadaecea89dd8254653559cc500f550b43c24390b8983600010193d2a4 not found: ID does not exist"
Sep 17 21:04:35.974589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.974477    2586 scope.go:117] "RemoveContainer" containerID="f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32"
Sep 17 21:04:35.974727 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:04:35.974708    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32\": container with ID starting with f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32 not found: ID does not exist" containerID="f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32"
Sep 17 21:04:35.974786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.974733    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32"} err="failed to get container status \"f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32\": rpc error: code = NotFound desc = could not find container \"f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32\": container with ID starting with f162625118c2f653d14e51771b2c06174abc79ffeb06b603b4564f8552a68f32 not found: ID does not exist"
Sep 17 21:04:35.974786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.974747    2586 scope.go:117] "RemoveContainer" containerID="66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9"
Sep 17 21:04:35.975002 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:04:35.974981    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9\": container with ID starting with 66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9 not found: ID does not exist" containerID="66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9"
Sep 17 21:04:35.975074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.975003    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9"} err="failed to get container status \"66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9\": rpc error: code = NotFound desc = could not find container \"66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9\": container with ID starting with 66cdc5c099aa04288e5266729535f966c1a9b5720b7b937698cba10d13916bd9 not found: ID does not exist"
Sep 17 21:04:35.975074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.975017    2586 scope.go:117] "RemoveContainer" containerID="5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da"
Sep 17 21:04:35.975254 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:04:35.975238    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da\": container with ID starting with 5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da not found: ID does not exist" containerID="5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da"
Sep 17 21:04:35.975289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.975256    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da"} err="failed to get container status \"5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da\": rpc error: code = NotFound desc = could not find container \"5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da\": container with ID starting with 5a746e2467097a873bc22892147c68fc81433b1c91025a69c0b7a03510bda5da not found: ID does not exist"
Sep 17 21:04:35.985192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.985173    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"]
Sep 17 21:04:35.991535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:35.991506    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5dccb4b654-dx5pt"]
Sep 17 21:04:36.670861 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:36.670829    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" path="/var/lib/kubelet/pods/44d18304-38f8-44cf-bf9d-cee1c4687a5c/volumes"
Sep 17 21:04:36.949411 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:36.949325    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:36.950556 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:36.950532    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:04:37.931220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:37.931196    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:04:37.931875 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:37.931838    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:04:37.932052 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:37.932029    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:04:37.953100 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:37.953073    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:04:42.957447 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:42.957416    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:04:42.958077 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:42.958050    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:04:46.582202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:46.582178    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:04:46.583570 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:46.583550    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:04:47.932697 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:47.932654    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:04:47.933126 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:47.933094    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:04:52.958677 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:52.958644    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:04:57.932522 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:57.932482    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:04:57.933019 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:04:57.932986    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:05:02.958498 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:02.958456    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:05:07.931872 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:07.931829    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:05:07.932348 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:07.932309    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:05:12.958562 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:12.958521    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:05:17.931847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:17.931810    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:05:17.932326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:17.932305    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:05:22.958384 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:22.958328    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:05:27.932140 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:27.932097    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:05:27.932686 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:27.932659    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:05:32.958874 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:32.958839    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:05:37.932683 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:37.932641    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:05:37.935073 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:37.933061    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:05:42.958928 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:42.958889    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:05:45.667565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:45.667532    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:05:45.667940 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:45.667767    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:05:52.959412 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:05:52.959385    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:06:00.596232 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.596150    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-9whlx_19c7695e-b061-4d60-8187-4198cff4441e/kserve-container/0.log"
Sep 17 21:06:00.791851 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.791819    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"]
Sep 17 21:06:00.792113 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.792064    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" containerID="cri-o://37526268fd9f1e30ecfc1a3b7f11f155ba76070a53b9ba8308c98692d2871da0" gracePeriod=30
Sep 17 21:06:00.792249 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.792115    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" containerID="cri-o://ba23ccfbc1a1a19c626749ed1ed89bbd8bb11e9d679a098369ac22908a2d0f61" gracePeriod=30
Sep 17 21:06:00.792249 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.792124    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy" containerID="cri-o://21e48d47396c3c1cb33e846dc724774ef82a05b56cc137d37b9fee3c965aa801" gracePeriod=30
Sep 17 21:06:00.884088 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.884048    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"]
Sep 17 21:06:00.884370 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.884311    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kserve-container" containerID="cri-o://9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da" gracePeriod=30
Sep 17 21:06:00.884482 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.884333    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kube-rbac-proxy" containerID="cri-o://6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa" gracePeriod=30
Sep 17 21:06:00.890786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.890762    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"]
Sep 17 21:06:00.891068 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891056    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container"
Sep 17 21:06:00.891110 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891072    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container"
Sep 17 21:06:00.891110 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891089    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy"
Sep 17 21:06:00.891110 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891096    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy"
Sep 17 21:06:00.891110 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891104    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent"
Sep 17 21:06:00.891110 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891109    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent"
Sep 17 21:06:00.891329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891117    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="storage-initializer"
Sep 17 21:06:00.891329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891123    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="storage-initializer"
Sep 17 21:06:00.891329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891176    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kserve-container"
Sep 17 21:06:00.891329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891183    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="kube-rbac-proxy"
Sep 17 21:06:00.891329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.891192    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d18304-38f8-44cf-bf9d-cee1c4687a5c" containerName="agent"
Sep 17 21:06:00.894339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.894311    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:00.896471 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.896430    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 21:06:00.896542 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.896473    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 21:06:00.902893 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:00.902874    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"]
Sep 17 21:06:01.009955 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.009930    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64272dfb-c2ae-4cdc-bab5-920c58dfb608-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.010049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.009979    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqcv2\" (UniqueName: \"kubernetes.io/projected/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-api-access-mqcv2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.010049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.010003    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.010049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.010037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.010186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.010057    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.110895 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.110859    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64272dfb-c2ae-4cdc-bab5-920c58dfb608-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.111020 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.110915    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqcv2\" (UniqueName: \"kubernetes.io/projected/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-api-access-mqcv2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.111020 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.110947    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.111020 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.110973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.111020 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.110999    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.111779 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.111709    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.111922 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.111812    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.112197 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.112174    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.113813 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.113783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64272dfb-c2ae-4cdc-bab5-920c58dfb608-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.119030 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.119010    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqcv2\" (UniqueName: \"kubernetes.io/projected/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-api-access-mqcv2\") pod \"isvc-autogluon-v1-predictor-574b5b7495-qhcm7\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.131608 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.131590    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:06:01.211687 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.211610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c7695e-b061-4d60-8187-4198cff4441e-proxy-tls\") pod \"19c7695e-b061-4d60-8187-4198cff4441e\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") "
Sep 17 21:06:01.211833 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.211690    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gx7d\" (UniqueName: \"kubernetes.io/projected/19c7695e-b061-4d60-8187-4198cff4441e-kube-api-access-7gx7d\") pod \"19c7695e-b061-4d60-8187-4198cff4441e\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") "
Sep 17 21:06:01.211833 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.211732    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c7695e-b061-4d60-8187-4198cff4441e-kube-rbac-proxy-sar-config\") pod \"19c7695e-b061-4d60-8187-4198cff4441e\" (UID: \"19c7695e-b061-4d60-8187-4198cff4441e\") "
Sep 17 21:06:01.212222 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.212193    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c7695e-b061-4d60-8187-4198cff4441e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19c7695e-b061-4d60-8187-4198cff4441e" (UID: "19c7695e-b061-4d60-8187-4198cff4441e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:01.215326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.215300    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c7695e-b061-4d60-8187-4198cff4441e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19c7695e-b061-4d60-8187-4198cff4441e" (UID: "19c7695e-b061-4d60-8187-4198cff4441e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:06:01.215455 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.215332    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c7695e-b061-4d60-8187-4198cff4441e-kube-api-access-7gx7d" (OuterVolumeSpecName: "kube-api-access-7gx7d") pod "19c7695e-b061-4d60-8187-4198cff4441e" (UID: "19c7695e-b061-4d60-8187-4198cff4441e"). InnerVolumeSpecName "kube-api-access-7gx7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:06:01.232996 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.232970    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"]
Sep 17 21:06:01.233213 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.233189    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" containerID="cri-o://dadf588d834a59be7d0e446635117732f97aef87521385630d00eef63496ba68" gracePeriod=30
Sep 17 21:06:01.233274 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.233253    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kube-rbac-proxy" containerID="cri-o://9a6ec940293822ff01c7a781593dad49b428b9ed873b7ca8efa3439f768d5fee" gracePeriod=30
Sep 17 21:06:01.256373 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.256329    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:06:01.263512 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.263485    2586 generic.go:356] "Generic (PLEG): container finished" podID="19c7695e-b061-4d60-8187-4198cff4441e" containerID="6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa" exitCode=2
Sep 17 21:06:01.263512 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.263507    2586 generic.go:356] "Generic (PLEG): container finished" podID="19c7695e-b061-4d60-8187-4198cff4441e" containerID="9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da" exitCode=2
Sep 17 21:06:01.263664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.263523    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"
Sep 17 21:06:01.263664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.263571    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" event={"ID":"19c7695e-b061-4d60-8187-4198cff4441e","Type":"ContainerDied","Data":"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"}
Sep 17 21:06:01.263664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.263624    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" event={"ID":"19c7695e-b061-4d60-8187-4198cff4441e","Type":"ContainerDied","Data":"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"}
Sep 17 21:06:01.263664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.263654    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx" event={"ID":"19c7695e-b061-4d60-8187-4198cff4441e","Type":"ContainerDied","Data":"6a8b5228f6b31f695895623c2dfd4cafb828b98fa96d4f660b056120c90bad61"}
Sep 17 21:06:01.263857 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.263684    2586 scope.go:117] "RemoveContainer" containerID="6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"
Sep 17 21:06:01.266129 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.266106    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3d27677-8197-461d-891e-73308d02f186" containerID="21e48d47396c3c1cb33e846dc724774ef82a05b56cc137d37b9fee3c965aa801" exitCode=2
Sep 17 21:06:01.266223 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.266149    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerDied","Data":"21e48d47396c3c1cb33e846dc724774ef82a05b56cc137d37b9fee3c965aa801"}
Sep 17 21:06:01.285032 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.285005    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"]
Sep 17 21:06:01.289185 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.289166    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9whlx"]
Sep 17 21:06:01.313944 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.313294    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c7695e-b061-4d60-8187-4198cff4441e-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:01.313944 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.313325    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gx7d\" (UniqueName: \"kubernetes.io/projected/19c7695e-b061-4d60-8187-4198cff4441e-kube-api-access-7gx7d\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:01.313944 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.313344    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c7695e-b061-4d60-8187-4198cff4441e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:01.343852 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.343831    2586 scope.go:117] "RemoveContainer" containerID="9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"
Sep 17 21:06:01.350789 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.350769    2586 scope.go:117] "RemoveContainer" containerID="6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"
Sep 17 21:06:01.351079 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:06:01.351059    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa\": container with ID starting with 6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa not found: ID does not exist" containerID="6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"
Sep 17 21:06:01.351130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.351090    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"} err="failed to get container status \"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa\": rpc error: code = NotFound desc = could not find container \"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa\": container with ID starting with 6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa not found: ID does not exist"
Sep 17 21:06:01.351130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.351107    2586 scope.go:117] "RemoveContainer" containerID="9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"
Sep 17 21:06:01.351340 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:06:01.351326    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da\": container with ID starting with 9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da not found: ID does not exist" containerID="9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"
Sep 17 21:06:01.351400 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.351344    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"} err="failed to get container status \"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da\": rpc error: code = NotFound desc = could not find container \"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da\": container with ID starting with 9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da not found: ID does not exist"
Sep 17 21:06:01.351400 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.351379    2586 scope.go:117] "RemoveContainer" containerID="6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"
Sep 17 21:06:01.351640 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.351620    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa"} err="failed to get container status \"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa\": rpc error: code = NotFound desc = could not find container \"6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa\": container with ID starting with 6222f82c277e39251409eeadd69c9bc555fa22de5cf1227617ff440e505b63fa not found: ID does not exist"
Sep 17 21:06:01.351712 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.351643    2586 scope.go:117] "RemoveContainer" containerID="9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"
Sep 17 21:06:01.351916 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.351899    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da"} err="failed to get container status \"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da\": rpc error: code = NotFound desc = could not find container \"9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da\": container with ID starting with 9143e0103ff723efef61df49fe67d457f79c97055bab6e48c92c8347272186da not found: ID does not exist"
Sep 17 21:06:01.538368 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:01.538329    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"]
Sep 17 21:06:01.540047 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:06:01.540024    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64272dfb_c2ae_4cdc_bab5_920c58dfb608.slice/crio-9b0b59d032d1a308833d9cad52705031a98ce01537f1dabe829a58b3ebc42076 WatchSource:0}: Error finding container 9b0b59d032d1a308833d9cad52705031a98ce01537f1dabe829a58b3ebc42076: Status 404 returned error can't find the container with id 9b0b59d032d1a308833d9cad52705031a98ce01537f1dabe829a58b3ebc42076
Sep 17 21:06:02.272286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.272253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerStarted","Data":"97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5"}
Sep 17 21:06:02.272743 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.272322    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerStarted","Data":"9b0b59d032d1a308833d9cad52705031a98ce01537f1dabe829a58b3ebc42076"}
Sep 17 21:06:02.274397 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.274376    2586 generic.go:356] "Generic (PLEG): container finished" podID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerID="9a6ec940293822ff01c7a781593dad49b428b9ed873b7ca8efa3439f768d5fee" exitCode=2
Sep 17 21:06:02.274451 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.274424    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerDied","Data":"9a6ec940293822ff01c7a781593dad49b428b9ed873b7ca8efa3439f768d5fee"}
Sep 17 21:06:02.671368 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.671324    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c7695e-b061-4d60-8187-4198cff4441e" path="/var/lib/kubelet/pods/19c7695e-b061-4d60-8187-4198cff4441e/volumes"
Sep 17 21:06:02.927546 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.927459    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 21:06:02.953664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.953635    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 21:06:02.958988 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:02.958965    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 21:06:05.667794 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:05.667750    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:06:05.668196 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:05.668062    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:06:06.292042 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.292014    2586 generic.go:356] "Generic (PLEG): container finished" podID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerID="dadf588d834a59be7d0e446635117732f97aef87521385630d00eef63496ba68" exitCode=0
Sep 17 21:06:06.292186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.292087    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerDied","Data":"dadf588d834a59be7d0e446635117732f97aef87521385630d00eef63496ba68"}
Sep 17 21:06:06.294523 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.294499    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3d27677-8197-461d-891e-73308d02f186" containerID="37526268fd9f1e30ecfc1a3b7f11f155ba76070a53b9ba8308c98692d2871da0" exitCode=0
Sep 17 21:06:06.294632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.294557    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerDied","Data":"37526268fd9f1e30ecfc1a3b7f11f155ba76070a53b9ba8308c98692d2871da0"}
Sep 17 21:06:06.466026 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.466002    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:06:06.555703 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.555674    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-cabundle-cert\") pod \"e5c789ea-c59b-435f-acbb-02d693325eb7\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") "
Sep 17 21:06:06.555703 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.555705    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5c789ea-c59b-435f-acbb-02d693325eb7-proxy-tls\") pod \"e5c789ea-c59b-435f-acbb-02d693325eb7\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") "
Sep 17 21:06:06.555889 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.555732    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-rbac-proxy-sar-config\") pod \"e5c789ea-c59b-435f-acbb-02d693325eb7\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") "
Sep 17 21:06:06.555889 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.555763    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jt5z\" (UniqueName: \"kubernetes.io/projected/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-api-access-5jt5z\") pod \"e5c789ea-c59b-435f-acbb-02d693325eb7\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") "
Sep 17 21:06:06.555889 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.555796    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5c789ea-c59b-435f-acbb-02d693325eb7-kserve-provision-location\") pod \"e5c789ea-c59b-435f-acbb-02d693325eb7\" (UID: \"e5c789ea-c59b-435f-acbb-02d693325eb7\") "
Sep 17 21:06:06.556089 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.556060    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5c789ea-c59b-435f-acbb-02d693325eb7" (UID: "e5c789ea-c59b-435f-acbb-02d693325eb7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:06.556179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.556063    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5c789ea-c59b-435f-acbb-02d693325eb7" (UID: "e5c789ea-c59b-435f-acbb-02d693325eb7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:06.556256 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.556238    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5c789ea-c59b-435f-acbb-02d693325eb7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5c789ea-c59b-435f-acbb-02d693325eb7" (UID: "e5c789ea-c59b-435f-acbb-02d693325eb7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:06.557751 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.557731    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5c789ea-c59b-435f-acbb-02d693325eb7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5c789ea-c59b-435f-acbb-02d693325eb7" (UID: "e5c789ea-c59b-435f-acbb-02d693325eb7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:06:06.557829 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.557778    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-api-access-5jt5z" (OuterVolumeSpecName: "kube-api-access-5jt5z") pod "e5c789ea-c59b-435f-acbb-02d693325eb7" (UID: "e5c789ea-c59b-435f-acbb-02d693325eb7"). InnerVolumeSpecName "kube-api-access-5jt5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:06:06.657282 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.657225    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:06.657282 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.657249    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jt5z\" (UniqueName: \"kubernetes.io/projected/e5c789ea-c59b-435f-acbb-02d693325eb7-kube-api-access-5jt5z\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:06.657282 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.657262    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5c789ea-c59b-435f-acbb-02d693325eb7-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:06.657282 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.657274    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5c789ea-c59b-435f-acbb-02d693325eb7-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:06.657497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:06.657290    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5c789ea-c59b-435f-acbb-02d693325eb7-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:07.300281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.300251    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"
Sep 17 21:06:07.300281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.300258    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl" event={"ID":"e5c789ea-c59b-435f-acbb-02d693325eb7","Type":"ContainerDied","Data":"0d42d49363b62778c1f227b85c093a88828590e3b41a9f3d375542cd40e55e92"}
Sep 17 21:06:07.300817 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.300324    2586 scope.go:117] "RemoveContainer" containerID="9a6ec940293822ff01c7a781593dad49b428b9ed873b7ca8efa3439f768d5fee"
Sep 17 21:06:07.308211 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.308190    2586 scope.go:117] "RemoveContainer" containerID="dadf588d834a59be7d0e446635117732f97aef87521385630d00eef63496ba68"
Sep 17 21:06:07.314731 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.314713    2586 scope.go:117] "RemoveContainer" containerID="f0a4ede9c00b4f4c8ce459a5db3bc366b59fdf8afef5f1680cfc0d0508a07976"
Sep 17 21:06:07.318890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.318870    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"]
Sep 17 21:06:07.322159 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.322141    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-62vhl"]
Sep 17 21:06:07.927748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:07.927710    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 21:06:08.670249 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:08.670215    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" path="/var/lib/kubelet/pods/e5c789ea-c59b-435f-acbb-02d693325eb7/volumes"
Sep 17 21:06:12.927729 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:12.927688    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 21:06:12.928118 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:12.927834    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:06:15.667517 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:15.667472    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:06:15.667996 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:15.667792    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:06:17.927451 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:17.927410    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 21:06:22.927847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:22.927805    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 21:06:25.667227 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:25.667177    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 21:06:25.667636 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:25.667305    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:06:25.667636 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:25.667388    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 21:06:25.667636 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:25.667495    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:06:27.927786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:27.927746    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 21:06:31.390344 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.390313    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3d27677-8197-461d-891e-73308d02f186" containerID="ba23ccfbc1a1a19c626749ed1ed89bbd8bb11e9d679a098369ac22908a2d0f61" exitCode=0
Sep 17 21:06:31.390740 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.390385    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerDied","Data":"ba23ccfbc1a1a19c626749ed1ed89bbd8bb11e9d679a098369ac22908a2d0f61"}
Sep 17 21:06:31.392051 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.392034    2586 generic.go:356] "Generic (PLEG): container finished" podID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerID="97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5" exitCode=0
Sep 17 21:06:31.392168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.392108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerDied","Data":"97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5"}
Sep 17 21:06:31.434604 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.434587    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:06:31.435029 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.435014    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-cabundle-cert\") pod \"d3d27677-8197-461d-891e-73308d02f186\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") "
Sep 17 21:06:31.435071 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.435042    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85mm6\" (UniqueName: \"kubernetes.io/projected/d3d27677-8197-461d-891e-73308d02f186-kube-api-access-85mm6\") pod \"d3d27677-8197-461d-891e-73308d02f186\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") "
Sep 17 21:06:31.435105 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.435074    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-kube-rbac-proxy-sar-config\") pod \"d3d27677-8197-461d-891e-73308d02f186\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") "
Sep 17 21:06:31.435105 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.435099    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d27677-8197-461d-891e-73308d02f186-kserve-provision-location\") pod \"d3d27677-8197-461d-891e-73308d02f186\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") "
Sep 17 21:06:31.435536 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.435512    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3d27677-8197-461d-891e-73308d02f186" (UID: "d3d27677-8197-461d-891e-73308d02f186"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:31.435632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.435528    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d27677-8197-461d-891e-73308d02f186-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3d27677-8197-461d-891e-73308d02f186" (UID: "d3d27677-8197-461d-891e-73308d02f186"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:06:31.435722 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.435704    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3d27677-8197-461d-891e-73308d02f186" (UID: "d3d27677-8197-461d-891e-73308d02f186"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:06:31.437074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.437053    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d27677-8197-461d-891e-73308d02f186-kube-api-access-85mm6" (OuterVolumeSpecName: "kube-api-access-85mm6") pod "d3d27677-8197-461d-891e-73308d02f186" (UID: "d3d27677-8197-461d-891e-73308d02f186"). InnerVolumeSpecName "kube-api-access-85mm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:06:31.536104 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.536064    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d27677-8197-461d-891e-73308d02f186-proxy-tls\") pod \"d3d27677-8197-461d-891e-73308d02f186\" (UID: \"d3d27677-8197-461d-891e-73308d02f186\") "
Sep 17 21:06:31.536246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.536181    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:31.536246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.536192    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85mm6\" (UniqueName: \"kubernetes.io/projected/d3d27677-8197-461d-891e-73308d02f186-kube-api-access-85mm6\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:31.536246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.536202    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d27677-8197-461d-891e-73308d02f186-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:31.536246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.536211    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d27677-8197-461d-891e-73308d02f186-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:31.537961 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.537939    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d27677-8197-461d-891e-73308d02f186-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3d27677-8197-461d-891e-73308d02f186" (UID: "d3d27677-8197-461d-891e-73308d02f186"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:06:31.637303 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:31.637278    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d27677-8197-461d-891e-73308d02f186-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:06:32.399807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:32.399770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48" event={"ID":"d3d27677-8197-461d-891e-73308d02f186","Type":"ContainerDied","Data":"9b0d944da44a0a77773538e04a0130dcdab45067101161fd00d40b86db360e12"}
Sep 17 21:06:32.400281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:32.399865    2586 scope.go:117] "RemoveContainer" containerID="ba23ccfbc1a1a19c626749ed1ed89bbd8bb11e9d679a098369ac22908a2d0f61"
Sep 17 21:06:32.400281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:32.399889    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"
Sep 17 21:06:32.426787 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:32.426758    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"]
Sep 17 21:06:32.433876 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:32.433854    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-8559c4556c-drp48"]
Sep 17 21:06:32.671196 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:32.671119    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d27677-8197-461d-891e-73308d02f186" path="/var/lib/kubelet/pods/d3d27677-8197-461d-891e-73308d02f186/volumes"
Sep 17 21:06:33.591766 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:33.591550    2586 scope.go:117] "RemoveContainer" containerID="21e48d47396c3c1cb33e846dc724774ef82a05b56cc137d37b9fee3c965aa801"
Sep 17 21:06:33.690011 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:33.689992    2586 scope.go:117] "RemoveContainer" containerID="37526268fd9f1e30ecfc1a3b7f11f155ba76070a53b9ba8308c98692d2871da0"
Sep 17 21:06:33.787975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:06:33.787952    2586 scope.go:117] "RemoveContainer" containerID="b5c01a77f0e63896216fb295c70f3cde95a3d970086e9177881688081c61051a"
Sep 17 21:08:36.891801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:36.891765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerStarted","Data":"dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712"}
Sep 17 21:08:36.892254 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:36.891827    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerStarted","Data":"357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160"}
Sep 17 21:08:36.892254 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:36.891921    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:08:36.892254 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:36.891978    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:08:36.893350 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:36.893328    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 21:08:36.913016 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:36.912973    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" podStartSLOduration=3.055928648 podStartE2EDuration="2m36.912959297s" totalImagesPullingTime="2m4.484885129s" totalInitContainerRuntime="29.37214552s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:08:36.911027424 +0000 UTC m=+1132.444967233" watchObservedRunningTime="2026-09-17 21:08:36.912959297 +0000 UTC m=+1132.446899106"
Sep 17 21:08:37.896411 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:37.896379    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 21:08:42.900537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:42.900505    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:08:42.900978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:42.900941    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 21:08:52.902178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:08:52.902143    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:09:06.989648 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:06.989565    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"]
Sep 17 21:09:06.990047 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:06.989911    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kserve-container" containerID="cri-o://357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160" gracePeriod=30
Sep 17 21:09:06.990047 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:06.989931    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kube-rbac-proxy" containerID="cri-o://dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712" gracePeriod=30
Sep 17 21:09:07.896822 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:07.896786    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 17 21:09:08.008958 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.008925    2586 generic.go:356] "Generic (PLEG): container finished" podID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerID="dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712" exitCode=2
Sep 17 21:09:08.009335 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.008966    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerDied","Data":"dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712"}
Sep 17 21:09:08.455878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.455855    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:09:08.524224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.524159    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqcv2\" (UniqueName: \"kubernetes.io/projected/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-api-access-mqcv2\") pod \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") "
Sep 17 21:09:08.524224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.524213    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kserve-provision-location\") pod \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") "
Sep 17 21:09:08.524501 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.524235    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-cabundle-cert\") pod \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") "
Sep 17 21:09:08.524501 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.524256    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64272dfb-c2ae-4cdc-bab5-920c58dfb608-proxy-tls\") pod \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") "
Sep 17 21:09:08.524501 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.524276    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-rbac-proxy-sar-config\") pod \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\" (UID: \"64272dfb-c2ae-4cdc-bab5-920c58dfb608\") "
Sep 17 21:09:08.524665 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.524619    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64272dfb-c2ae-4cdc-bab5-920c58dfb608" (UID: "64272dfb-c2ae-4cdc-bab5-920c58dfb608"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:08.524721 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.524670    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "64272dfb-c2ae-4cdc-bab5-920c58dfb608" (UID: "64272dfb-c2ae-4cdc-bab5-920c58dfb608"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:09:08.526339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.526316    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-api-access-mqcv2" (OuterVolumeSpecName: "kube-api-access-mqcv2") pod "64272dfb-c2ae-4cdc-bab5-920c58dfb608" (UID: "64272dfb-c2ae-4cdc-bab5-920c58dfb608"). InnerVolumeSpecName "kube-api-access-mqcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:09:08.526465 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.526333    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64272dfb-c2ae-4cdc-bab5-920c58dfb608-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "64272dfb-c2ae-4cdc-bab5-920c58dfb608" (UID: "64272dfb-c2ae-4cdc-bab5-920c58dfb608"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:09:08.528849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.528823    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64272dfb-c2ae-4cdc-bab5-920c58dfb608" (UID: "64272dfb-c2ae-4cdc-bab5-920c58dfb608"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:09:08.624840 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.624801    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqcv2\" (UniqueName: \"kubernetes.io/projected/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-api-access-mqcv2\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:09:08.624840 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.624834    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:09:08.624840 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.624846    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:09:08.625084 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.624858    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/64272dfb-c2ae-4cdc-bab5-920c58dfb608-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:09:08.625084 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:08.624869    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/64272dfb-c2ae-4cdc-bab5-920c58dfb608-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:09:09.014178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.014146    2586 generic.go:356] "Generic (PLEG): container finished" podID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerID="357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160" exitCode=0
Sep 17 21:09:09.014670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.014202    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"
Sep 17 21:09:09.014670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.014225    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerDied","Data":"357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160"}
Sep 17 21:09:09.014670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.014283    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7" event={"ID":"64272dfb-c2ae-4cdc-bab5-920c58dfb608","Type":"ContainerDied","Data":"9b0b59d032d1a308833d9cad52705031a98ce01537f1dabe829a58b3ebc42076"}
Sep 17 21:09:09.014670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.014309    2586 scope.go:117] "RemoveContainer" containerID="dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712"
Sep 17 21:09:09.021906 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.021891    2586 scope.go:117] "RemoveContainer" containerID="357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160"
Sep 17 21:09:09.028990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.028973    2586 scope.go:117] "RemoveContainer" containerID="97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5"
Sep 17 21:09:09.032623 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.032606    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"]
Sep 17 21:09:09.036286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.036271    2586 scope.go:117] "RemoveContainer" containerID="dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712"
Sep 17 21:09:09.036616 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:09:09.036596    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712\": container with ID starting with dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712 not found: ID does not exist" containerID="dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712"
Sep 17 21:09:09.036678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.036632    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712"} err="failed to get container status \"dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712\": rpc error: code = NotFound desc = could not find container \"dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712\": container with ID starting with dea407498419f88ece379095449e585bccd271b01cc179372471c7a0836c4712 not found: ID does not exist"
Sep 17 21:09:09.036678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.036650    2586 scope.go:117] "RemoveContainer" containerID="357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160"
Sep 17 21:09:09.036968 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:09:09.036951    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160\": container with ID starting with 357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160 not found: ID does not exist" containerID="357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160"
Sep 17 21:09:09.037019 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.036972    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160"} err="failed to get container status \"357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160\": rpc error: code = NotFound desc = could not find container \"357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160\": container with ID starting with 357cb0639fa86550739066e6cb91476ef43ca3045ed9b048473f520e2aadf160 not found: ID does not exist"
Sep 17 21:09:09.037019 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.036986    2586 scope.go:117] "RemoveContainer" containerID="97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5"
Sep 17 21:09:09.037227 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:09:09.037210    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5\": container with ID starting with 97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5 not found: ID does not exist" containerID="97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5"
Sep 17 21:09:09.037279 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.037234    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5"} err="failed to get container status \"97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5\": rpc error: code = NotFound desc = could not find container \"97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5\": container with ID starting with 97d77913d52c6224117b1e3e6a1a3be914a0aa4ea5179fa29f01305f17e261d5 not found: ID does not exist"
Sep 17 21:09:09.037489 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:09.037472    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-qhcm7"]
Sep 17 21:09:10.670568 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:10.670530    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" path="/var/lib/kubelet/pods/64272dfb-c2ae-4cdc-bab5-920c58dfb608/volumes"
Sep 17 21:09:12.245670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.245628    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"]
Sep 17 21:09:12.246058 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246042    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="storage-initializer"
Sep 17 21:09:12.246102 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246060    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="storage-initializer"
Sep 17 21:09:12.246102 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246072    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246102 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246081    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246102 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246093    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246103    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246112    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="storage-initializer"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246120    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="storage-initializer"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246135    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246144    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246154    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246162    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246172    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="storage-initializer"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246180    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="storage-initializer"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246189    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kserve-container"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246197    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kserve-container"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246208    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246215    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246225    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246234    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246245    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246253    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246262    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246270    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246325    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246335    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246343    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246367    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246375    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246384    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="64272dfb-c2ae-4cdc-bab5-920c58dfb608" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246392    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c7695e-b061-4d60-8187-4198cff4441e" containerName="kube-rbac-proxy"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246401    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c789ea-c59b-435f-acbb-02d693325eb7" containerName="kserve-container"
Sep 17 21:09:12.246600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.246408    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d27677-8197-461d-891e-73308d02f186" containerName="agent"
Sep 17 21:09:12.274086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.274055    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"]
Sep 17 21:09:12.274266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.274188    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.276950 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.276929    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 21:09:12.276950 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.276945    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:09:12.277141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.276929    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:09:12.277295 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.277279    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:09:12.277942 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.277926    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:09:12.278046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.277980    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 21:09:12.278046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.277987    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:09:12.351374 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.351327    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52684254-4f90-49d9-9068-f6fd20818da4-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.351514 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.351390    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpzg\" (UniqueName: \"kubernetes.io/projected/52684254-4f90-49d9-9068-f6fd20818da4-kube-api-access-btpzg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.351514 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.351421    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52684254-4f90-49d9-9068-f6fd20818da4-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.351514 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.351449    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.351514 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.351465    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.452570 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.452541    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52684254-4f90-49d9-9068-f6fd20818da4-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.452718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.452587    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpzg\" (UniqueName: \"kubernetes.io/projected/52684254-4f90-49d9-9068-f6fd20818da4-kube-api-access-btpzg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.452718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.452611    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52684254-4f90-49d9-9068-f6fd20818da4-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.452718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.452652    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.452718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.452679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.452994 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.452973    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52684254-4f90-49d9-9068-f6fd20818da4-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.453283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.453265    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.453377 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.453292    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.455088 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.455066    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52684254-4f90-49d9-9068-f6fd20818da4-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.461304 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.461277    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpzg\" (UniqueName: \"kubernetes.io/projected/52684254-4f90-49d9-9068-f6fd20818da4-kube-api-access-btpzg\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-9t7q2\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.584000 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.583921    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:12.874992 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:12.874947    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"]
Sep 17 21:09:12.877098 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:09:12.877065    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52684254_4f90_49d9_9068_f6fd20818da4.slice/crio-f97b8b40f52fbed35bc5656658673d21492e43c7a97b20a23e872b34e8b4cf09 WatchSource:0}: Error finding container f97b8b40f52fbed35bc5656658673d21492e43c7a97b20a23e872b34e8b4cf09: Status 404 returned error can't find the container with id f97b8b40f52fbed35bc5656658673d21492e43c7a97b20a23e872b34e8b4cf09
Sep 17 21:09:13.031086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:13.031051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerStarted","Data":"50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5"}
Sep 17 21:09:13.031236 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:13.031108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerStarted","Data":"f97b8b40f52fbed35bc5656658673d21492e43c7a97b20a23e872b34e8b4cf09"}
Sep 17 21:09:18.049951 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:18.049916    2586 generic.go:356] "Generic (PLEG): container finished" podID="52684254-4f90-49d9-9068-f6fd20818da4" containerID="50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5" exitCode=0
Sep 17 21:09:18.050442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:18.049996    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerDied","Data":"50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5"}
Sep 17 21:09:19.056553 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:19.056522    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerStarted","Data":"e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73"}
Sep 17 21:09:19.056909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:19.056583    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerStarted","Data":"28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a"}
Sep 17 21:09:19.056909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:19.056842    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:19.075712 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:19.075666    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" podStartSLOduration=2.205201401 podStartE2EDuration="7.075651441s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.87045004s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:09:19.074633286 +0000 UTC m=+1174.608573086" watchObservedRunningTime="2026-09-17 21:09:19.075651441 +0000 UTC m=+1174.609591250"
Sep 17 21:09:20.060802 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:20.060774    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:26.070903 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:26.070871    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:09:46.605389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:46.605347    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:09:46.605932 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:09:46.605914    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:10:51.699398 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:51.699350    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:10:58.242179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:58.242146    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"]
Sep 17 21:10:58.242594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:58.242391    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kserve-container" containerID="cri-o://28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a" gracePeriod=30
Sep 17 21:10:58.242594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:58.242458    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kube-rbac-proxy" containerID="cri-o://e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73" gracePeriod=30
Sep 17 21:10:58.408957 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:58.408928    2586 generic.go:356] "Generic (PLEG): container finished" podID="52684254-4f90-49d9-9068-f6fd20818da4" containerID="e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73" exitCode=2
Sep 17 21:10:58.409058 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:58.408962    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerDied","Data":"e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73"}
Sep 17 21:10:59.575281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.575258    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:10:59.585956 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.585934    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52684254-4f90-49d9-9068-f6fd20818da4-proxy-tls\") pod \"52684254-4f90-49d9-9068-f6fd20818da4\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") "
Sep 17 21:10:59.586054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.585979    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52684254-4f90-49d9-9068-f6fd20818da4-kserve-provision-location\") pod \"52684254-4f90-49d9-9068-f6fd20818da4\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") "
Sep 17 21:10:59.586054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.586011    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-cabundle-cert\") pod \"52684254-4f90-49d9-9068-f6fd20818da4\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") "
Sep 17 21:10:59.586130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.586056    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btpzg\" (UniqueName: \"kubernetes.io/projected/52684254-4f90-49d9-9068-f6fd20818da4-kube-api-access-btpzg\") pod \"52684254-4f90-49d9-9068-f6fd20818da4\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") "
Sep 17 21:10:59.586130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.586088    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-kube-rbac-proxy-sar-config\") pod \"52684254-4f90-49d9-9068-f6fd20818da4\" (UID: \"52684254-4f90-49d9-9068-f6fd20818da4\") "
Sep 17 21:10:59.586427 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.586401    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52684254-4f90-49d9-9068-f6fd20818da4" (UID: "52684254-4f90-49d9-9068-f6fd20818da4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:10:59.586523 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.586479    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52684254-4f90-49d9-9068-f6fd20818da4" (UID: "52684254-4f90-49d9-9068-f6fd20818da4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:10:59.587962 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.587942    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52684254-4f90-49d9-9068-f6fd20818da4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52684254-4f90-49d9-9068-f6fd20818da4" (UID: "52684254-4f90-49d9-9068-f6fd20818da4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:10:59.588130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.588103    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52684254-4f90-49d9-9068-f6fd20818da4-kube-api-access-btpzg" (OuterVolumeSpecName: "kube-api-access-btpzg") pod "52684254-4f90-49d9-9068-f6fd20818da4" (UID: "52684254-4f90-49d9-9068-f6fd20818da4"). InnerVolumeSpecName "kube-api-access-btpzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:10:59.590847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.590828    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52684254-4f90-49d9-9068-f6fd20818da4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52684254-4f90-49d9-9068-f6fd20818da4" (UID: "52684254-4f90-49d9-9068-f6fd20818da4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:10:59.686576 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.686507    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpzg\" (UniqueName: \"kubernetes.io/projected/52684254-4f90-49d9-9068-f6fd20818da4-kube-api-access-btpzg\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:10:59.686576 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.686534    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:10:59.686576 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.686547    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52684254-4f90-49d9-9068-f6fd20818da4-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:10:59.686576 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.686559    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52684254-4f90-49d9-9068-f6fd20818da4-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:10:59.686576 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:10:59.686573    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52684254-4f90-49d9-9068-f6fd20818da4-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:11:00.418431 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.418394    2586 generic.go:356] "Generic (PLEG): container finished" podID="52684254-4f90-49d9-9068-f6fd20818da4" containerID="28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a" exitCode=0
Sep 17 21:11:00.418652 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.418462    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"
Sep 17 21:11:00.418652 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.418473    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerDied","Data":"28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a"}
Sep 17 21:11:00.418652 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.418547    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2" event={"ID":"52684254-4f90-49d9-9068-f6fd20818da4","Type":"ContainerDied","Data":"f97b8b40f52fbed35bc5656658673d21492e43c7a97b20a23e872b34e8b4cf09"}
Sep 17 21:11:00.418652 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.418585    2586 scope.go:117] "RemoveContainer" containerID="e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73"
Sep 17 21:11:00.426603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.426589    2586 scope.go:117] "RemoveContainer" containerID="28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a"
Sep 17 21:11:00.433550 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.433532    2586 scope.go:117] "RemoveContainer" containerID="50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5"
Sep 17 21:11:00.440745 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.440698    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"]
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.440899    2586 scope.go:117] "RemoveContainer" containerID="e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:11:00.441318    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73\": container with ID starting with e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73 not found: ID does not exist" containerID="e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.441368    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73"} err="failed to get container status \"e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73\": rpc error: code = NotFound desc = could not find container \"e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73\": container with ID starting with e3ed3a4208f63046d4262701fb6009ea4d40a311d2dec300ec20d8a37a905b73 not found: ID does not exist"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.441394    2586 scope.go:117] "RemoveContainer" containerID="28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:11:00.442258    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a\": container with ID starting with 28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a not found: ID does not exist" containerID="28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.442314    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a"} err="failed to get container status \"28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a\": rpc error: code = NotFound desc = could not find container \"28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a\": container with ID starting with 28a68f5202d7ba7f75664b7a02c7ccd06d38f3eb35398545f3221668372cfe3a not found: ID does not exist"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.442344    2586 scope.go:117] "RemoveContainer" containerID="50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:11:00.443636    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5\": container with ID starting with 50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5 not found: ID does not exist" containerID="50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.443664    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5"} err="failed to get container status \"50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5\": rpc error: code = NotFound desc = could not find container \"50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5\": container with ID starting with 50e149af7182b6f8e4378bc205a72c5d7144870d2184d2629f69a0a0943010f5 not found: ID does not exist"
Sep 17 21:11:00.445883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.445141    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-9t7q2"]
Sep 17 21:11:00.670908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:00.670828    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52684254-4f90-49d9-9068-f6fd20818da4" path="/var/lib/kubelet/pods/52684254-4f90-49d9-9068-f6fd20818da4/volumes"
Sep 17 21:11:03.459282 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459251    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"]
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459512    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kserve-container"
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459525    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kserve-container"
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459537    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="storage-initializer"
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459543    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="storage-initializer"
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459551    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kube-rbac-proxy"
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459557    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kube-rbac-proxy"
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459597    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kserve-container"
Sep 17 21:11:03.459649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.459603    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="52684254-4f90-49d9-9068-f6fd20818da4" containerName="kube-rbac-proxy"
Sep 17 21:11:03.462624 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.462605    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.465188 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.465170    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:11:03.465290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.465239    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:11:03.465400 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.465376    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:11:03.465520 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.465440    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 21:11:03.465520 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.465466    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:11:03.465649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.465564    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:11:03.466126 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.466111    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 21:11:03.473225 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.473205    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"]
Sep 17 21:11:03.510815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.510792    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhwb\" (UniqueName: \"kubernetes.io/projected/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-api-access-nkhwb\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.510914 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.510818    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.510914 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.510838    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.510914 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.510882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.511014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.510927    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.611777 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.611745    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.611897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.611786    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.611897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.611832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhwb\" (UniqueName: \"kubernetes.io/projected/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-api-access-nkhwb\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.611897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.611849    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.611897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.611866    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.612080 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:11:03.611986    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 21:11:03.612080 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:11:03.612056    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls podName:f11fd609-7ec6-4e6a-b3f1-19062830b1f9 nodeName:}" failed. No retries permitted until 2026-09-17 21:11:04.112033137 +0000 UTC m=+1279.645972924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" (UID: "f11fd609-7ec6-4e6a-b3f1-19062830b1f9") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 17 21:11:03.612174 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.612131    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.612604 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.612584    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.612648 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.612584    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:03.622524 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:03.622507    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhwb\" (UniqueName: \"kubernetes.io/projected/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-api-access-nkhwb\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:04.115535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:04.115501    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:04.117723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:04.117702    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:04.372524 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:04.372440    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:04.639999 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:04.639975    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"]
Sep 17 21:11:04.641580 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:11:04.641545    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11fd609_7ec6_4e6a_b3f1_19062830b1f9.slice/crio-13aa5e555c96d95c5f3e66e23bbb2698b69fc1f7d599be004f33426a026809d7 WatchSource:0}: Error finding container 13aa5e555c96d95c5f3e66e23bbb2698b69fc1f7d599be004f33426a026809d7: Status 404 returned error can't find the container with id 13aa5e555c96d95c5f3e66e23bbb2698b69fc1f7d599be004f33426a026809d7
Sep 17 21:11:05.439214 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:05.439176    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerStarted","Data":"9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a"}
Sep 17 21:11:05.439396 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:05.439250    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerStarted","Data":"13aa5e555c96d95c5f3e66e23bbb2698b69fc1f7d599be004f33426a026809d7"}
Sep 17 21:11:10.459370 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:10.459322    2586 generic.go:356] "Generic (PLEG): container finished" podID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerID="9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a" exitCode=0
Sep 17 21:11:10.459789 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:10.459395    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerDied","Data":"9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a"}
Sep 17 21:11:11.465451 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:11.465418    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerStarted","Data":"4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba"}
Sep 17 21:11:11.465924 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:11.465488    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerStarted","Data":"f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16"}
Sep 17 21:11:11.465924 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:11.465678    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:11.484713 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:11.484664    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" podStartSLOduration=3.616545179 podStartE2EDuration="8.484650395s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.868105216s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:11:11.48381291 +0000 UTC m=+1287.017752719" watchObservedRunningTime="2026-09-17 21:11:11.484650395 +0000 UTC m=+1287.018590204"
Sep 17 21:11:12.469514 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:12.469476    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:11:18.478655 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:11:18.478629    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:12:06.976003 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:06.975929    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"]
Sep 17 21:12:06.978939 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:06.978923    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:06.982685 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:06.982664    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 21:12:06.986858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:06.986840    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 21:12:06.997138 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:06.997116    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"]
Sep 17 21:12:07.046956 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.046930    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-proxy-tls\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.047055 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.046960    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kserve-provision-location\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.047055 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.046987    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.047055 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.047046    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdsb7\" (UniqueName: \"kubernetes.io/projected/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-api-access-xdsb7\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.047160 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.047071    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-cabundle-cert\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.147373 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.147333    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdsb7\" (UniqueName: \"kubernetes.io/projected/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-api-access-xdsb7\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.147478 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.147385    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-cabundle-cert\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.147478 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.147418    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-proxy-tls\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.147478 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.147433    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kserve-provision-location\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.147478 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.147461    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.147854 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.147831    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kserve-provision-location\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.148164 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.148144    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-cabundle-cert\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.148211 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.148159    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.149908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.149889    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-proxy-tls\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.158407 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.158389    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdsb7\" (UniqueName: \"kubernetes.io/projected/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-api-access-xdsb7\") pod \"isvc-sklearn-predictor-bb6f97488-dddjk\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.288466 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.288410    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:07.558974 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.558949    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"]
Sep 17 21:12:07.561291 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:12:07.561265    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34fc7e5_6d9a_4b0e_a8e1_5a2c26cb7892.slice/crio-d7784be3847a5870021de6609575317e5e1df1278a3a3e1956d727fdaec8b17f WatchSource:0}: Error finding container d7784be3847a5870021de6609575317e5e1df1278a3a3e1956d727fdaec8b17f: Status 404 returned error can't find the container with id d7784be3847a5870021de6609575317e5e1df1278a3a3e1956d727fdaec8b17f
Sep 17 21:12:07.668405 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.668370    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerStarted","Data":"1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28"}
Sep 17 21:12:07.668557 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:07.668430    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerStarted","Data":"d7784be3847a5870021de6609575317e5e1df1278a3a3e1956d727fdaec8b17f"}
Sep 17 21:12:11.682264 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:11.682232    2586 generic.go:356] "Generic (PLEG): container finished" podID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerID="1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28" exitCode=0
Sep 17 21:12:11.682610 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:11.682305    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerDied","Data":"1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28"}
Sep 17 21:12:12.687532 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:12.687495    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerStarted","Data":"484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70"}
Sep 17 21:12:12.687894 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:12.687555    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerStarted","Data":"2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b"}
Sep 17 21:12:12.687894 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:12.687788    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:12.716068 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:12.716020    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podStartSLOduration=2.800535324 podStartE2EDuration="6.716006907s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.915471583s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:12:12.714483203 +0000 UTC m=+1348.248423014" watchObservedRunningTime="2026-09-17 21:12:12.716006907 +0000 UTC m=+1348.249946716"
Sep 17 21:12:13.690782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:13.690755    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:13.692066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:13.692033    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:12:14.693941 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:14.693898    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:12:19.698699 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:19.698666    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:12:19.699334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:19.699305    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:12:29.699607 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:29.699570    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:12:39.700031 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:39.699986    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:12:46.706288 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:46.706258    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:12:49.699618 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:49.699585    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:12:57.713525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:57.713476    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 21:12:59.700049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:59.700002    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:12:59.943912 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:59.943881    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"]
Sep 17 21:12:59.944196 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:59.944155    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kserve-container" containerID="cri-o://f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16" gracePeriod=30
Sep 17 21:12:59.944292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:12:59.944176    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kube-rbac-proxy" containerID="cri-o://4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba" gracePeriod=30
Sep 17 21:13:00.864244 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:00.864209    2586 generic.go:356] "Generic (PLEG): container finished" podID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerID="4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba" exitCode=2
Sep 17 21:13:00.864844 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:00.864285    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerDied","Data":"4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba"}
Sep 17 21:13:01.277187 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.277162    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:13:01.327310 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.327285    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkhwb\" (UniqueName: \"kubernetes.io/projected/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-api-access-nkhwb\") pod \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") "
Sep 17 21:13:01.327505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.327318    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-cabundle-cert\") pod \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") "
Sep 17 21:13:01.327505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.327346    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kserve-provision-location\") pod \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") "
Sep 17 21:13:01.327505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.327387    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-rbac-proxy-sar-config\") pod \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") "
Sep 17 21:13:01.327505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.327424    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls\") pod \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\" (UID: \"f11fd609-7ec6-4e6a-b3f1-19062830b1f9\") "
Sep 17 21:13:01.327735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.327715    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f11fd609-7ec6-4e6a-b3f1-19062830b1f9" (UID: "f11fd609-7ec6-4e6a-b3f1-19062830b1f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:13:01.327796 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.327759    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f11fd609-7ec6-4e6a-b3f1-19062830b1f9" (UID: "f11fd609-7ec6-4e6a-b3f1-19062830b1f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:13:01.329428 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.329405    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f11fd609-7ec6-4e6a-b3f1-19062830b1f9" (UID: "f11fd609-7ec6-4e6a-b3f1-19062830b1f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:13:01.329537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.329454    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-api-access-nkhwb" (OuterVolumeSpecName: "kube-api-access-nkhwb") pod "f11fd609-7ec6-4e6a-b3f1-19062830b1f9" (UID: "f11fd609-7ec6-4e6a-b3f1-19062830b1f9"). InnerVolumeSpecName "kube-api-access-nkhwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:13:01.332436 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.332410    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f11fd609-7ec6-4e6a-b3f1-19062830b1f9" (UID: "f11fd609-7ec6-4e6a-b3f1-19062830b1f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:13:01.427963 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.427901    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkhwb\" (UniqueName: \"kubernetes.io/projected/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-api-access-nkhwb\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:13:01.427963 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.427926    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:13:01.427963 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.427940    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:13:01.427963 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.427954    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:13:01.428158 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.427967    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f11fd609-7ec6-4e6a-b3f1-19062830b1f9-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:13:01.870242 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.870207    2586 generic.go:356] "Generic (PLEG): container finished" podID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerID="f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16" exitCode=0
Sep 17 21:13:01.870664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.870257    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"
Sep 17 21:13:01.870664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.870287    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerDied","Data":"f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16"}
Sep 17 21:13:01.870664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.870341    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x" event={"ID":"f11fd609-7ec6-4e6a-b3f1-19062830b1f9","Type":"ContainerDied","Data":"13aa5e555c96d95c5f3e66e23bbb2698b69fc1f7d599be004f33426a026809d7"}
Sep 17 21:13:01.870664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.870384    2586 scope.go:117] "RemoveContainer" containerID="4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba"
Sep 17 21:13:01.879132 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.879118    2586 scope.go:117] "RemoveContainer" containerID="f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16"
Sep 17 21:13:01.888252 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.888236    2586 scope.go:117] "RemoveContainer" containerID="9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a"
Sep 17 21:13:01.894612 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.894587    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"]
Sep 17 21:13:01.896316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.896283    2586 scope.go:117] "RemoveContainer" containerID="4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba"
Sep 17 21:13:01.896761 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:13:01.896729    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba\": container with ID starting with 4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba not found: ID does not exist" containerID="4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba"
Sep 17 21:13:01.896852 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.896767    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba"} err="failed to get container status \"4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba\": rpc error: code = NotFound desc = could not find container \"4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba\": container with ID starting with 4e705726447995bdae003c5e4d065d6ac1780308580a4da3b9a97728f265bfba not found: ID does not exist"
Sep 17 21:13:01.896852 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.896787    2586 scope.go:117] "RemoveContainer" containerID="f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16"
Sep 17 21:13:01.897074 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:13:01.897056    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16\": container with ID starting with f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16 not found: ID does not exist" containerID="f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16"
Sep 17 21:13:01.897134 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.897081    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16"} err="failed to get container status \"f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16\": rpc error: code = NotFound desc = could not find container \"f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16\": container with ID starting with f557049485b0ac86c216fc34cf155218ad08c2113077a917e2df04e4948a5b16 not found: ID does not exist"
Sep 17 21:13:01.897134 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.897098    2586 scope.go:117] "RemoveContainer" containerID="9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a"
Sep 17 21:13:01.897373 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:13:01.897341    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a\": container with ID starting with 9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a not found: ID does not exist" containerID="9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a"
Sep 17 21:13:01.897513 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.897377    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a"} err="failed to get container status \"9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a\": rpc error: code = NotFound desc = could not find container \"9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a\": container with ID starting with 9caab8a7e2c7f6b4b3ca8ce870e8ce91a6706182858c18ff6d08bd32dc87572a not found: ID does not exist"
Sep 17 21:13:01.897904 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:01.897886    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-klb2x"]
Sep 17 21:13:02.670861 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:02.670820    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" path="/var/lib/kubelet/pods/f11fd609-7ec6-4e6a-b3f1-19062830b1f9/volumes"
Sep 17 21:13:09.699171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:09.699140    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:13:19.699393 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:19.699333    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:13:22.666368 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:22.666327    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 21:13:32.670124 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:32.670094    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:13:37.297681 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:37.297641    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:39.316337 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:39.316311    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:41.326722 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:41.326693    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:43.337092 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:43.337064    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:45.347924 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:45.347897    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:47.358121 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:47.358089    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:49.372126 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:49.372099    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:51.382767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:51.382737    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:53.391695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:53.391659    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:55.402502 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:55.402476    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:57.412576 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:57.412543    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:13:59.422941 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:13:59.422906    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:14:01.433556 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:01.433529    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:14:03.444246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:03.444218    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:14:05.455825 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:05.455797    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:14:07.467005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:07.466942    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-predictor-bb6f97488-dddjk_a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/kserve-container/0.log"
Sep 17 21:14:07.584720 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:07.584689    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"]
Sep 17 21:14:07.584994 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:07.584955    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container" containerID="cri-o://2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b" gracePeriod=30
Sep 17 21:14:07.585074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:07.585009    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kube-rbac-proxy" containerID="cri-o://484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70" gracePeriod=30
Sep 17 21:14:08.104066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:08.104031    2586 generic.go:356] "Generic (PLEG): container finished" podID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerID="484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70" exitCode=2
Sep 17 21:14:08.104231 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:08.104100    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerDied","Data":"484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70"}
Sep 17 21:14:09.694347 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:09.694313    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 21:14:12.024143 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.024119    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:14:12.116035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116010    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-cabundle-cert\") pod \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") "
Sep 17 21:14:12.116186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116062    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-rbac-proxy-sar-config\") pod \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") "
Sep 17 21:14:12.116186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116086    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdsb7\" (UniqueName: \"kubernetes.io/projected/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-api-access-xdsb7\") pod \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") "
Sep 17 21:14:12.116186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116115    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-proxy-tls\") pod \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") "
Sep 17 21:14:12.116186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116170    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kserve-provision-location\") pod \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\" (UID: \"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892\") "
Sep 17 21:14:12.116555 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116522    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" (UID: "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:14:12.116555 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116535    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" (UID: "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:14:12.116692 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.116564    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" (UID: "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:14:12.122936 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.122841    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" (UID: "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:14:12.122936 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.122907    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-api-access-xdsb7" (OuterVolumeSpecName: "kube-api-access-xdsb7") pod "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" (UID: "a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892"). InnerVolumeSpecName "kube-api-access-xdsb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:14:12.125037 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.124992    2586 generic.go:356] "Generic (PLEG): container finished" podID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerID="2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b" exitCode=0
Sep 17 21:14:12.125153 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.125123    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerDied","Data":"2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b"}
Sep 17 21:14:12.125241 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.125136    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"
Sep 17 21:14:12.125295 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.125251    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk" event={"ID":"a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892","Type":"ContainerDied","Data":"d7784be3847a5870021de6609575317e5e1df1278a3a3e1956d727fdaec8b17f"}
Sep 17 21:14:12.125364 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.125300    2586 scope.go:117] "RemoveContainer" containerID="484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70"
Sep 17 21:14:12.133602 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.133589    2586 scope.go:117] "RemoveContainer" containerID="2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b"
Sep 17 21:14:12.140198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.140184    2586 scope.go:117] "RemoveContainer" containerID="1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28"
Sep 17 21:14:12.146456 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.146347    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"]
Sep 17 21:14:12.147068 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.147054    2586 scope.go:117] "RemoveContainer" containerID="484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70"
Sep 17 21:14:12.147393 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:14:12.147375    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70\": container with ID starting with 484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70 not found: ID does not exist" containerID="484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70"
Sep 17 21:14:12.147442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.147401    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70"} err="failed to get container status \"484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70\": rpc error: code = NotFound desc = could not find container \"484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70\": container with ID starting with 484aea7bf5e1ae0a00dc945e18d57ce65d90499621bf205fb42599305be16b70 not found: ID does not exist"
Sep 17 21:14:12.147442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.147418    2586 scope.go:117] "RemoveContainer" containerID="2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b"
Sep 17 21:14:12.147684 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:14:12.147667    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b\": container with ID starting with 2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b not found: ID does not exist" containerID="2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b"
Sep 17 21:14:12.147727 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.147688    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b"} err="failed to get container status \"2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b\": rpc error: code = NotFound desc = could not find container \"2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b\": container with ID starting with 2c941e841a78438eaa0ab53551364d9115c230914013f9158518fd281e21bb5b not found: ID does not exist"
Sep 17 21:14:12.147727 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.147703    2586 scope.go:117] "RemoveContainer" containerID="1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28"
Sep 17 21:14:12.147957 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:14:12.147942    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28\": container with ID starting with 1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28 not found: ID does not exist" containerID="1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28"
Sep 17 21:14:12.148012 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.147961    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28"} err="failed to get container status \"1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28\": rpc error: code = NotFound desc = could not find container \"1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28\": container with ID starting with 1aa99e9f211292747b582836169a0e727f444627fb42edf6128c2341e89ffe28 not found: ID does not exist"
Sep 17 21:14:12.150791 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.150771    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-bb6f97488-dddjk"]
Sep 17 21:14:12.217142 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.217124    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:14:12.217142 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.217143    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:14:12.217272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.217153    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:14:12.217272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.217163    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:14:12.217272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.217171    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdsb7\" (UniqueName: \"kubernetes.io/projected/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892-kube-api-access-xdsb7\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:14:12.671517 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.671482    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" path="/var/lib/kubelet/pods/a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892/volumes"
Sep 17 21:14:12.824188 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824159    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"]
Sep 17 21:14:12.824412 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824401    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="storage-initializer"
Sep 17 21:14:12.824463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824414    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="storage-initializer"
Sep 17 21:14:12.824463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824429    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kube-rbac-proxy"
Sep 17 21:14:12.824463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824435    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kube-rbac-proxy"
Sep 17 21:14:12.824463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824441    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kube-rbac-proxy"
Sep 17 21:14:12.824463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824446    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kube-rbac-proxy"
Sep 17 21:14:12.824463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824457    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="storage-initializer"
Sep 17 21:14:12.824463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824462    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="storage-initializer"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824471    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kserve-container"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824476    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kserve-container"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824482    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824486    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824526    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kserve-container"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824533    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kserve-container"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824539    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f11fd609-7ec6-4e6a-b3f1-19062830b1f9" containerName="kube-rbac-proxy"
Sep 17 21:14:12.824671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.824545    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34fc7e5-6d9a-4b0e-a8e1-5a2c26cb7892" containerName="kube-rbac-proxy"
Sep 17 21:14:12.829177 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.829157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:12.832017 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.831987    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:14:12.832130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.832041    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:14:12.832341 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.832327    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:14:12.832482 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.832468    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:14:12.833191 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.833175    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 21:14:12.833265 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.833176    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:14:12.833500 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.833483    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 21:14:12.839973 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.839953    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"]
Sep 17 21:14:12.921780 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.921711    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:12.921780 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.921754    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46b54a-7dd9-44c5-ab93-05f584256c3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:12.921780 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.921777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:12.922022 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.921800    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqgh\" (UniqueName: \"kubernetes.io/projected/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-api-access-vnqgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:12.922022 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:12.921877    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023106 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023250 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023122    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023250 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023154    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46b54a-7dd9-44c5-ab93-05f584256c3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023395 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023276    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023395 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023319    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqgh\" (UniqueName: \"kubernetes.io/projected/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-api-access-vnqgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023546 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023520    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023863 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023838    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.023863 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.023857    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.025596 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.025575    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46b54a-7dd9-44c5-ab93-05f584256c3d-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.031340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.031320    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqgh\" (UniqueName: \"kubernetes.io/projected/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-api-access-vnqgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.139636 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.139611    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:14:13.407763 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:13.407736    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"]
Sep 17 21:14:13.409755 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:14:13.409725    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa46b54a_7dd9_44c5_ab93_05f584256c3d.slice/crio-e192fa8f9ea84f2e3c7b33bafaf8b3f813753d416465e4ff5fef12235d2c0028 WatchSource:0}: Error finding container e192fa8f9ea84f2e3c7b33bafaf8b3f813753d416465e4ff5fef12235d2c0028: Status 404 returned error can't find the container with id e192fa8f9ea84f2e3c7b33bafaf8b3f813753d416465e4ff5fef12235d2c0028
Sep 17 21:14:14.136267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:14.136232    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerStarted","Data":"47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8"}
Sep 17 21:14:14.136791 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:14.136300    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerStarted","Data":"e192fa8f9ea84f2e3c7b33bafaf8b3f813753d416465e4ff5fef12235d2c0028"}
Sep 17 21:14:18.151009 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:18.150976    2586 generic.go:356] "Generic (PLEG): container finished" podID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerID="47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8" exitCode=0
Sep 17 21:14:18.151412 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:18.151046    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerDied","Data":"47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8"}
Sep 17 21:14:18.152175 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:18.152157    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:14:46.630419 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:46.630390    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:14:46.632274 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:14:46.632247    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:16:26.688689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:16:26.688653    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerStarted","Data":"cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a"}
Sep 17 21:16:26.689171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:16:26.688709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerStarted","Data":"c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a"}
Sep 17 21:16:26.689171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:16:26.688861    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:16:26.689171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:16:26.688921    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:16:26.711922 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:16:26.711872    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" podStartSLOduration=2.7668675289999998 podStartE2EDuration="2m14.711852673s" totalImagesPullingTime="2m8.074321208s" totalInitContainerRuntime="3.870663936s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:16:26.71153394 +0000 UTC m=+1602.245473748" watchObservedRunningTime="2026-09-17 21:16:26.711852673 +0000 UTC m=+1602.245792484"
Sep 17 21:16:32.698293 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:16:32.697890    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:17:02.711755 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:02.711716    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 21:17:12.700888 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:12.700859    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:17:23.021490 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:23.021414    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"]
Sep 17 21:17:23.021876 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:23.021643    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kserve-container" containerID="cri-o://c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a" gracePeriod=30
Sep 17 21:17:23.021876 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:23.021696    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kube-rbac-proxy" containerID="cri-o://cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a" gracePeriod=30
Sep 17 21:17:23.887575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:23.887544    2586 generic.go:356] "Generic (PLEG): container finished" podID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerID="cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a" exitCode=2
Sep 17 21:17:23.887741 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:23.887613    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerDied","Data":"cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a"}
Sep 17 21:17:27.693203 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:27.693157    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 21:17:28.164226 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.164186    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"]
Sep 17 21:17:28.167062 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.167044    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.169385 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.169338    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 21:17:28.169557 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.169541    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:17:28.176814 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.176793    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"]
Sep 17 21:17:28.289491 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.289458    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a018379e-5ef7-4b74-b61a-ab422b748e9f-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.289640 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.289495    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a018379e-5ef7-4b74-b61a-ab422b748e9f-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.289640 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.289528    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.289640 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.289626    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmx9t\" (UniqueName: \"kubernetes.io/projected/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-api-access-lmx9t\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.289754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.289653    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.390321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.390291    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a018379e-5ef7-4b74-b61a-ab422b748e9f-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.390321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.390322    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a018379e-5ef7-4b74-b61a-ab422b748e9f-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.390535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.390349    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.390535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.390396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmx9t\" (UniqueName: \"kubernetes.io/projected/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-api-access-lmx9t\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.390535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.390412    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.390808 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.390784    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a018379e-5ef7-4b74-b61a-ab422b748e9f-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.391133 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.391116    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.391194 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.391141    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.392799 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.392777    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a018379e-5ef7-4b74-b61a-ab422b748e9f-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.409966 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.409939    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmx9t\" (UniqueName: \"kubernetes.io/projected/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-api-access-lmx9t\") pod \"isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.478099 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.478032    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:28.748636 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.748612    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"]
Sep 17 21:17:28.751288 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:17:28.751263    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda018379e_5ef7_4b74_b61a_ab422b748e9f.slice/crio-914cf05974663e5f01ade41e86f66ef4ec8312008758da9383169b908e125551 WatchSource:0}: Error finding container 914cf05974663e5f01ade41e86f66ef4ec8312008758da9383169b908e125551: Status 404 returned error can't find the container with id 914cf05974663e5f01ade41e86f66ef4ec8312008758da9383169b908e125551
Sep 17 21:17:28.906816 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.906777    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerStarted","Data":"88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc"}
Sep 17 21:17:28.906972 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:28.906846    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerStarted","Data":"914cf05974663e5f01ade41e86f66ef4ec8312008758da9383169b908e125551"}
Sep 17 21:17:29.653171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.653150    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:17:29.801059 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.800994    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-rbac-proxy-sar-config\") pod \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") "
Sep 17 21:17:29.801059 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.801034    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46b54a-7dd9-44c5-ab93-05f584256c3d-proxy-tls\") pod \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") "
Sep 17 21:17:29.801521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.801058    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kserve-provision-location\") pod \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") "
Sep 17 21:17:29.801521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.801096    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-cabundle-cert\") pod \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") "
Sep 17 21:17:29.801521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.801145    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnqgh\" (UniqueName: \"kubernetes.io/projected/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-api-access-vnqgh\") pod \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\" (UID: \"aa46b54a-7dd9-44c5-ab93-05f584256c3d\") "
Sep 17 21:17:29.801521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.801306    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aa46b54a-7dd9-44c5-ab93-05f584256c3d" (UID: "aa46b54a-7dd9-44c5-ab93-05f584256c3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:17:29.801521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.801449    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aa46b54a-7dd9-44c5-ab93-05f584256c3d" (UID: "aa46b54a-7dd9-44c5-ab93-05f584256c3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:17:29.801521 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.801460    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aa46b54a-7dd9-44c5-ab93-05f584256c3d" (UID: "aa46b54a-7dd9-44c5-ab93-05f584256c3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:17:29.803042 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.803021    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa46b54a-7dd9-44c5-ab93-05f584256c3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa46b54a-7dd9-44c5-ab93-05f584256c3d" (UID: "aa46b54a-7dd9-44c5-ab93-05f584256c3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:17:29.803131 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.803111    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-api-access-vnqgh" (OuterVolumeSpecName: "kube-api-access-vnqgh") pod "aa46b54a-7dd9-44c5-ab93-05f584256c3d" (UID: "aa46b54a-7dd9-44c5-ab93-05f584256c3d"). InnerVolumeSpecName "kube-api-access-vnqgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:17:29.901617 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.901588    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnqgh\" (UniqueName: \"kubernetes.io/projected/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-api-access-vnqgh\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:17:29.901617 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.901614    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:17:29.901754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.901628    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa46b54a-7dd9-44c5-ab93-05f584256c3d-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:17:29.901754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.901639    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aa46b54a-7dd9-44c5-ab93-05f584256c3d-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:17:29.901754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.901653    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aa46b54a-7dd9-44c5-ab93-05f584256c3d-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:17:29.912663 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.912641    2586 generic.go:356] "Generic (PLEG): container finished" podID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerID="c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a" exitCode=0
Sep 17 21:17:29.912764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.912690    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"
Sep 17 21:17:29.912764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.912717    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerDied","Data":"c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a"}
Sep 17 21:17:29.912838 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.912771    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj" event={"ID":"aa46b54a-7dd9-44c5-ab93-05f584256c3d","Type":"ContainerDied","Data":"e192fa8f9ea84f2e3c7b33bafaf8b3f813753d416465e4ff5fef12235d2c0028"}
Sep 17 21:17:29.912838 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.912791    2586 scope.go:117] "RemoveContainer" containerID="cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a"
Sep 17 21:17:29.920620 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.920488    2586 scope.go:117] "RemoveContainer" containerID="c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a"
Sep 17 21:17:29.927487 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.927459    2586 scope.go:117] "RemoveContainer" containerID="47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8"
Sep 17 21:17:29.934938 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.934921    2586 scope.go:117] "RemoveContainer" containerID="cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a"
Sep 17 21:17:29.935234 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.935217    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"]
Sep 17 21:17:29.935274 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:17:29.935232    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a\": container with ID starting with cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a not found: ID does not exist" containerID="cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a"
Sep 17 21:17:29.935274 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.935253    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a"} err="failed to get container status \"cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a\": rpc error: code = NotFound desc = could not find container \"cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a\": container with ID starting with cb6a122de7f84fa32583c23297ee9d5051e160ce47ce1e00ee0753e92626269a not found: ID does not exist"
Sep 17 21:17:29.935274 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.935270    2586 scope.go:117] "RemoveContainer" containerID="c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a"
Sep 17 21:17:29.935567 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:17:29.935549    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a\": container with ID starting with c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a not found: ID does not exist" containerID="c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a"
Sep 17 21:17:29.935615 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.935571    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a"} err="failed to get container status \"c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a\": rpc error: code = NotFound desc = could not find container \"c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a\": container with ID starting with c505c8512edc84b70d62664b111b14a9b68dae93636cdde7fd94f04c00ae1f7a not found: ID does not exist"
Sep 17 21:17:29.935615 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.935587    2586 scope.go:117] "RemoveContainer" containerID="47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8"
Sep 17 21:17:29.935827 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:17:29.935812    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8\": container with ID starting with 47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8 not found: ID does not exist" containerID="47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8"
Sep 17 21:17:29.935866 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.935830    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8"} err="failed to get container status \"47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8\": rpc error: code = NotFound desc = could not find container \"47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8\": container with ID starting with 47f1283cdde1de11649b8263441169124c9b0352da68347139e11d2105eb57d8 not found: ID does not exist"
Sep 17 21:17:29.942949 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:29.942931    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-wqbzj"]
Sep 17 21:17:30.670447 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:30.670415    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" path="/var/lib/kubelet/pods/aa46b54a-7dd9-44c5-ab93-05f584256c3d/volumes"
Sep 17 21:17:35.935673 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:35.935642    2586 generic.go:356] "Generic (PLEG): container finished" podID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerID="88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc" exitCode=0
Sep 17 21:17:35.936067 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:35.935686    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerDied","Data":"88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc"}
Sep 17 21:17:36.941419 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:36.941383    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerStarted","Data":"aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a"}
Sep 17 21:17:36.941876 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:36.941454    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerStarted","Data":"b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce"}
Sep 17 21:17:36.941876 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:36.941666    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:36.961786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:36.961738    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" podStartSLOduration=2.624677653 podStartE2EDuration="8.961719479s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.337041826s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:17:36.960615302 +0000 UTC m=+1672.494555125" watchObservedRunningTime="2026-09-17 21:17:36.961719479 +0000 UTC m=+1672.495659289"
Sep 17 21:17:37.944905 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:37.944877    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:37.946180 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:37.946145    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 21:17:38.948741 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:38.948700    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 21:17:43.952973 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:43.952941    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:17:43.953611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:43.953574    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 21:17:53.954248 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:17:53.954217    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:18:00.713391 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:00.713347    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"]
Sep 17 21:18:00.713795 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:00.713576    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kserve-container" containerID="cri-o://b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce" gracePeriod=30
Sep 17 21:18:00.713795 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:00.713611    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kube-rbac-proxy" containerID="cri-o://aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a" gracePeriod=30
Sep 17 21:18:01.037901 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.037826    2586 generic.go:356] "Generic (PLEG): container finished" podID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerID="aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a" exitCode=2
Sep 17 21:18:01.038038 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.037896    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerDied","Data":"aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a"}
Sep 17 21:18:01.749985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.749959    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:18:01.943490 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943463    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-cabundle-cert\") pod \"a018379e-5ef7-4b74-b61a-ab422b748e9f\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") "
Sep 17 21:18:01.943675 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943508    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a018379e-5ef7-4b74-b61a-ab422b748e9f-kserve-provision-location\") pod \"a018379e-5ef7-4b74-b61a-ab422b748e9f\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") "
Sep 17 21:18:01.943675 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943557    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-rbac-proxy-sar-config\") pod \"a018379e-5ef7-4b74-b61a-ab422b748e9f\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") "
Sep 17 21:18:01.943807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943687    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a018379e-5ef7-4b74-b61a-ab422b748e9f-proxy-tls\") pod \"a018379e-5ef7-4b74-b61a-ab422b748e9f\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") "
Sep 17 21:18:01.943807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943746    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmx9t\" (UniqueName: \"kubernetes.io/projected/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-api-access-lmx9t\") pod \"a018379e-5ef7-4b74-b61a-ab422b748e9f\" (UID: \"a018379e-5ef7-4b74-b61a-ab422b748e9f\") "
Sep 17 21:18:01.943913 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943859    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a018379e-5ef7-4b74-b61a-ab422b748e9f" (UID: "a018379e-5ef7-4b74-b61a-ab422b748e9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:01.943913 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943883    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a018379e-5ef7-4b74-b61a-ab422b748e9f" (UID: "a018379e-5ef7-4b74-b61a-ab422b748e9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:18:01.943983 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943956    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:18:01.943983 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.943972    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:18:01.945753 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.945727    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-api-access-lmx9t" (OuterVolumeSpecName: "kube-api-access-lmx9t") pod "a018379e-5ef7-4b74-b61a-ab422b748e9f" (UID: "a018379e-5ef7-4b74-b61a-ab422b748e9f"). InnerVolumeSpecName "kube-api-access-lmx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:18:01.945845 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.945762    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a018379e-5ef7-4b74-b61a-ab422b748e9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a018379e-5ef7-4b74-b61a-ab422b748e9f" (UID: "a018379e-5ef7-4b74-b61a-ab422b748e9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:18:01.972401 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:01.972380    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a018379e-5ef7-4b74-b61a-ab422b748e9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a018379e-5ef7-4b74-b61a-ab422b748e9f" (UID: "a018379e-5ef7-4b74-b61a-ab422b748e9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:18:02.043551 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.043527    2586 generic.go:356] "Generic (PLEG): container finished" podID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerID="b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce" exitCode=0
Sep 17 21:18:02.043668 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.043586    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"
Sep 17 21:18:02.043668 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.043591    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerDied","Data":"b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce"}
Sep 17 21:18:02.043668 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.043656    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx" event={"ID":"a018379e-5ef7-4b74-b61a-ab422b748e9f","Type":"ContainerDied","Data":"914cf05974663e5f01ade41e86f66ef4ec8312008758da9383169b908e125551"}
Sep 17 21:18:02.043833 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.043694    2586 scope.go:117] "RemoveContainer" containerID="aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a"
Sep 17 21:18:02.044245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.044220    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmx9t\" (UniqueName: \"kubernetes.io/projected/a018379e-5ef7-4b74-b61a-ab422b748e9f-kube-api-access-lmx9t\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:18:02.044245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.044243    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a018379e-5ef7-4b74-b61a-ab422b748e9f-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:18:02.044385 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.044259    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a018379e-5ef7-4b74-b61a-ab422b748e9f-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:18:02.051994 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.051966    2586 scope.go:117] "RemoveContainer" containerID="b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce"
Sep 17 21:18:02.059248 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.059233    2586 scope.go:117] "RemoveContainer" containerID="88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc"
Sep 17 21:18:02.064572 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.064553    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"]
Sep 17 21:18:02.066172 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.066159    2586 scope.go:117] "RemoveContainer" containerID="aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a"
Sep 17 21:18:02.066468 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:18:02.066444    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a\": container with ID starting with aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a not found: ID does not exist" containerID="aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a"
Sep 17 21:18:02.066522 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.066497    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a"} err="failed to get container status \"aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a\": rpc error: code = NotFound desc = could not find container \"aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a\": container with ID starting with aa7a5ad661bd88ff766fad2ad3296dce21618c81ab0eaf0c99014ad67d74b67a not found: ID does not exist"
Sep 17 21:18:02.066584 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.066523    2586 scope.go:117] "RemoveContainer" containerID="b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce"
Sep 17 21:18:02.066782 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:18:02.066761    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce\": container with ID starting with b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce not found: ID does not exist" containerID="b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce"
Sep 17 21:18:02.066857 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.066785    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce"} err="failed to get container status \"b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce\": rpc error: code = NotFound desc = could not find container \"b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce\": container with ID starting with b68494a737200c2750c36b4e56b147948d3b7f0cac8fa395d7ba1a741b8ef9ce not found: ID does not exist"
Sep 17 21:18:02.066857 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.066798    2586 scope.go:117] "RemoveContainer" containerID="88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc"
Sep 17 21:18:02.067380 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:18:02.067146    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc\": container with ID starting with 88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc not found: ID does not exist" containerID="88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc"
Sep 17 21:18:02.067380 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.067178    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc"} err="failed to get container status \"88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc\": rpc error: code = NotFound desc = could not find container \"88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc\": container with ID starting with 88bf1f050a1772b595bf67d2a8a6257af68c7d8834cddd1ffd989ca021c2b5dc not found: ID does not exist"
Sep 17 21:18:02.068453 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.068435    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b87f46d9b-bpcfx"]
Sep 17 21:18:02.670762 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:02.670734    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" path="/var/lib/kubelet/pods/a018379e-5ef7-4b74-b61a-ab422b748e9f/volumes"
Sep 17 21:18:05.871259 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871226    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"]
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871459    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kserve-container"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871470    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kserve-container"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871481    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="storage-initializer"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871487    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="storage-initializer"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871497    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="storage-initializer"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871502    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="storage-initializer"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871509    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kube-rbac-proxy"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871514    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kube-rbac-proxy"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871519    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kube-rbac-proxy"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871525    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kube-rbac-proxy"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871537    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kserve-container"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871542    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kserve-container"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871581    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kube-rbac-proxy"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871589    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kserve-container"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871595    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46b54a-7dd9-44c5-ab93-05f584256c3d" containerName="kube-rbac-proxy"
Sep 17 21:18:05.871678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.871600    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a018379e-5ef7-4b74-b61a-ab422b748e9f" containerName="kserve-container"
Sep 17 21:18:05.874312 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.874295    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:05.877199 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.877181    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:18:05.878144 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.878117    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:18:05.878232 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.878158    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:18:05.878232 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.878199    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:18:05.878333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.878233    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 21:18:05.878333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.878204    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:18:05.878449 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.878414    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:18:05.889639 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.889620    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"]
Sep 17 21:18:05.969371 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.969327    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4zl\" (UniqueName: \"kubernetes.io/projected/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-api-access-hc4zl\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:05.969472 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.969391    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:05.969472 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.969415    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:05.969472 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.969444    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:05.969578 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:05.969500    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.070537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.070510    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.070735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.070555    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.070735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.070660    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.070735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.070715    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4zl\" (UniqueName: \"kubernetes.io/projected/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-api-access-hc4zl\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.070872 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.070746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.071021 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.070945    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.071285 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.071269    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.071346 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.071327    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.072954 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.072935    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.084035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.079783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4zl\" (UniqueName: \"kubernetes.io/projected/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-api-access-hc4zl\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.184098 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.184031    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:06.453139 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:06.453059    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"]
Sep 17 21:18:06.455760 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:18:06.455732    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3aa038_6f0d_43fb_b8fb_156cf9fdd44b.slice/crio-569a9ed88330c4e032291c1f1f9c3359d9d642fe44809a57e56ecc3c596f6845 WatchSource:0}: Error finding container 569a9ed88330c4e032291c1f1f9c3359d9d642fe44809a57e56ecc3c596f6845: Status 404 returned error can't find the container with id 569a9ed88330c4e032291c1f1f9c3359d9d642fe44809a57e56ecc3c596f6845
Sep 17 21:18:07.063335 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:07.063297    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerStarted","Data":"ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31"}
Sep 17 21:18:07.063708 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:07.063391    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerStarted","Data":"569a9ed88330c4e032291c1f1f9c3359d9d642fe44809a57e56ecc3c596f6845"}
Sep 17 21:18:11.078340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:11.078304    2586 generic.go:356] "Generic (PLEG): container finished" podID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerID="ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31" exitCode=0
Sep 17 21:18:11.078842 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:11.078387    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerDied","Data":"ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31"}
Sep 17 21:18:12.083995 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:12.083959    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerStarted","Data":"0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788"}
Sep 17 21:18:12.084415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:12.084046    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerStarted","Data":"8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec"}
Sep 17 21:18:12.084415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:12.084231    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:12.105808 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:12.105768    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" podStartSLOduration=3.306321677 podStartE2EDuration="7.105754272s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.799432595s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:18:12.104870075 +0000 UTC m=+1707.638809897" watchObservedRunningTime="2026-09-17 21:18:12.105754272 +0000 UTC m=+1707.639694080"
Sep 17 21:18:13.090174 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:13.090140    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:19.100688 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:19.100660    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:49.112113 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:49.112069    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 21:18:59.103036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.103004    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:18:59.511082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.511051    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"]
Sep 17 21:18:59.515648 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.515632    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.517900 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.517882    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 21:18:59.518240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.518225    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 21:18:59.525060 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.525033    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"]
Sep 17 21:18:59.552772 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.552746    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"]
Sep 17 21:18:59.555822 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.555806    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.559081 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.559066    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 21:18:59.568807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.568781    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"]
Sep 17 21:18:59.642108 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5164c1b-3b21-476e-9c92-8f5fda9ad063-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.642251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642114    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.642251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642140    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.642251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642227    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.642444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642260    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-api-access-jhwxz\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.642444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642323    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.642444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642348    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.642444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642404    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.642444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642433    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.642679 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.642465    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcsbf\" (UniqueName: \"kubernetes.io/projected/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-api-access-dcsbf\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.743091 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743062    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.743091 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743093    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.743285 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743113    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.743285 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743134    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-api-access-jhwxz\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.743285 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743160    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.743486 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743391    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.743486 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743412    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.743486 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743442    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.743486 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcsbf\" (UniqueName: \"kubernetes.io/projected/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-api-access-dcsbf\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.743723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743499    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5164c1b-3b21-476e-9c92-8f5fda9ad063-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.743723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743554    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.743723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.743612    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.744126 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.744095    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.744126 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.744115    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.744283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.744175    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.744283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.744197    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.745691 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.745669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-proxy-tls\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.745940 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.745923    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5164c1b-3b21-476e-9c92-8f5fda9ad063-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.754619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.754595    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-api-access-jhwxz\") pod \"isvc-canary-create-predictor-7c7f45c769-8cl4g\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:18:59.754702 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.754686    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcsbf\" (UniqueName: \"kubernetes.io/projected/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-api-access-dcsbf\") pod \"isvc-canary-create-v2-predictor-bfcdb48dd-hksnw\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.826653 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.826599    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:18:59.865429 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:18:59.865395    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:19:00.104459 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:00.102528    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"]
Sep 17 21:19:00.106225 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:19:00.106192    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0120be8b_5ad6_4d5c_a0ab_5eed89aa8f7a.slice/crio-9f1cd1e0d4eae27148718216046d60db137c2226861c2d00438d8e3ba036ebbb WatchSource:0}: Error finding container 9f1cd1e0d4eae27148718216046d60db137c2226861c2d00438d8e3ba036ebbb: Status 404 returned error can't find the container with id 9f1cd1e0d4eae27148718216046d60db137c2226861c2d00438d8e3ba036ebbb
Sep 17 21:19:00.143524 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:00.143500    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"]
Sep 17 21:19:00.147396 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:19:00.147334    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5164c1b_3b21_476e_9c92_8f5fda9ad063.slice/crio-79ed3f3053b217c925f42a0c7d9abead7ed8cd3fa6fc7a992d8aeca7440116d1 WatchSource:0}: Error finding container 79ed3f3053b217c925f42a0c7d9abead7ed8cd3fa6fc7a992d8aeca7440116d1: Status 404 returned error can't find the container with id 79ed3f3053b217c925f42a0c7d9abead7ed8cd3fa6fc7a992d8aeca7440116d1
Sep 17 21:19:00.255066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:00.255034    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerStarted","Data":"bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee"}
Sep 17 21:19:00.255255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:00.255104    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerStarted","Data":"79ed3f3053b217c925f42a0c7d9abead7ed8cd3fa6fc7a992d8aeca7440116d1"}
Sep 17 21:19:00.256702 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:00.256674    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerStarted","Data":"6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7"}
Sep 17 21:19:00.256828 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:00.256732    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerStarted","Data":"9f1cd1e0d4eae27148718216046d60db137c2226861c2d00438d8e3ba036ebbb"}
Sep 17 21:19:04.273056 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:04.272972    2586 generic.go:356] "Generic (PLEG): container finished" podID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerID="6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7" exitCode=0
Sep 17 21:19:04.273508 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:04.273054    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerDied","Data":"6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7"}
Sep 17 21:19:04.274718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:04.274695    2586 generic.go:356] "Generic (PLEG): container finished" podID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerID="bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee" exitCode=0
Sep 17 21:19:04.274803 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:04.274746    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerDied","Data":"bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee"}
Sep 17 21:19:05.280430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.280392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerStarted","Data":"b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86"}
Sep 17 21:19:05.280815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.280458    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerStarted","Data":"72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349"}
Sep 17 21:19:05.280815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.280736    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:19:05.280937 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.280888    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:19:05.282371 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.282326    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:19:05.282700 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.282682    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerStarted","Data":"cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9"}
Sep 17 21:19:05.282791 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.282735    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerStarted","Data":"c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2"}
Sep 17 21:19:05.282948 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.282932    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:19:05.315346 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.315303    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podStartSLOduration=2.586966719 podStartE2EDuration="6.315288786s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.728322067s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:05.314134746 +0000 UTC m=+1760.848074554" watchObservedRunningTime="2026-09-17 21:19:05.315288786 +0000 UTC m=+1760.849228651"
Sep 17 21:19:05.338477 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.338434    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podStartSLOduration=2.620834696 podStartE2EDuration="6.33842163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.717586934s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:05.337525366 +0000 UTC m=+1760.871465175" watchObservedRunningTime="2026-09-17 21:19:05.33842163 +0000 UTC m=+1760.872361439"
Sep 17 21:19:05.972800 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.972768    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"]
Sep 17 21:19:05.973035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.973014    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kserve-container" containerID="cri-o://8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec" gracePeriod=30
Sep 17 21:19:05.973117 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:05.973093    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kube-rbac-proxy" containerID="cri-o://0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788" gracePeriod=30
Sep 17 21:19:06.289152 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:06.289067    2586 generic.go:356] "Generic (PLEG): container finished" podID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerID="0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788" exitCode=2
Sep 17 21:19:06.289152 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:06.289138    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerDied","Data":"0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788"}
Sep 17 21:19:06.289607 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:06.289459    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:19:06.289607 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:06.289578    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:19:06.290801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:06.290776    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:19:07.293663 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:07.293626    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:19:09.094992 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:09.094945    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 21:19:11.161076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.161041    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"]
Sep 17 21:19:11.164440 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.164424    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.169038 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.169016    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 21:19:11.169154 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.169054    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 21:19:11.178866 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.178844    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"]
Sep 17 21:19:11.238053 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.238025    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.238239 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.238059    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.238239 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.238077    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pt9w\" (UniqueName: \"kubernetes.io/projected/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-api-access-7pt9w\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.238239 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.238179    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.238239 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.238224    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.294075 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.294053    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:19:11.294668 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.294642    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:19:11.338928 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.338898    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.339052 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.338942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.339105 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.339061    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.339144 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.339104    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.339144 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.339129    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pt9w\" (UniqueName: \"kubernetes.io/projected/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-api-access-7pt9w\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.339244 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:19:11.339220    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 21:19:11.339302 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.339260    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.339302 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:19:11.339294    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls podName:7c851cf5-ddaa-4fea-af35-958c7fc9f623 nodeName:}" failed. No retries permitted until 2026-09-17 21:19:11.839272735 +0000 UTC m=+1767.373212524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls") pod "isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" (UID: "7c851cf5-ddaa-4fea-af35-958c7fc9f623") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 21:19:11.339685 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.339669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.339735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.339687    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.350552 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.350534    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pt9w\" (UniqueName: \"kubernetes.io/projected/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-api-access-7pt9w\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.843762 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.843728    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:11.846046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:11.846027    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-hq7g7\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:12.074086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:12.074042    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:12.298686 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:12.298655    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:19:12.299112 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:12.299088    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:19:12.351649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:12.351612    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"]
Sep 17 21:19:12.353876 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:19:12.353850    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c851cf5_ddaa_4fea_af35_958c7fc9f623.slice/crio-718df4dabe52942e8335accd0825f9ad6f34a492423febcb50de03ed2478a090 WatchSource:0}: Error finding container 718df4dabe52942e8335accd0825f9ad6f34a492423febcb50de03ed2478a090: Status 404 returned error can't find the container with id 718df4dabe52942e8335accd0825f9ad6f34a492423febcb50de03ed2478a090
Sep 17 21:19:13.220114 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.220089    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:19:13.321580 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.321538    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerStarted","Data":"b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574"}
Sep 17 21:19:13.321971 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.321653    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerStarted","Data":"718df4dabe52942e8335accd0825f9ad6f34a492423febcb50de03ed2478a090"}
Sep 17 21:19:13.323517 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.323492    2586 generic.go:356] "Generic (PLEG): container finished" podID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerID="8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec" exitCode=0
Sep 17 21:19:13.323646 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.323537    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"
Sep 17 21:19:13.323646 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.323539    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerDied","Data":"8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec"}
Sep 17 21:19:13.323733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.323681    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w" event={"ID":"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b","Type":"ContainerDied","Data":"569a9ed88330c4e032291c1f1f9c3359d9d642fe44809a57e56ecc3c596f6845"}
Sep 17 21:19:13.323733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.323722    2586 scope.go:117] "RemoveContainer" containerID="0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788"
Sep 17 21:19:13.331198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.331181    2586 scope.go:117] "RemoveContainer" containerID="8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec"
Sep 17 21:19:13.338055 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.338036    2586 scope.go:117] "RemoveContainer" containerID="ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31"
Sep 17 21:19:13.345255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.345240    2586 scope.go:117] "RemoveContainer" containerID="0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788"
Sep 17 21:19:13.345532 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:19:13.345512    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788\": container with ID starting with 0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788 not found: ID does not exist" containerID="0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788"
Sep 17 21:19:13.345597 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.345539    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788"} err="failed to get container status \"0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788\": rpc error: code = NotFound desc = could not find container \"0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788\": container with ID starting with 0ed660a97536492064fdcf2506ad34dbad82c8fd3d22bc5f196a293e250f3788 not found: ID does not exist"
Sep 17 21:19:13.345597 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.345555    2586 scope.go:117] "RemoveContainer" containerID="8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec"
Sep 17 21:19:13.345794 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:19:13.345777    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec\": container with ID starting with 8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec not found: ID does not exist" containerID="8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec"
Sep 17 21:19:13.345837 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.345800    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec"} err="failed to get container status \"8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec\": rpc error: code = NotFound desc = could not find container \"8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec\": container with ID starting with 8c265202771e2eef02fd3753f3639b1fe81ae7a558a8af1f4ffd4ecaa728beec not found: ID does not exist"
Sep 17 21:19:13.345837 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.345815    2586 scope.go:117] "RemoveContainer" containerID="ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31"
Sep 17 21:19:13.346047 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:19:13.346027    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31\": container with ID starting with ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31 not found: ID does not exist" containerID="ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31"
Sep 17 21:19:13.346147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.346052    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31"} err="failed to get container status \"ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31\": rpc error: code = NotFound desc = could not find container \"ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31\": container with ID starting with ec619ad3e6aff90632e01e98fc8c20af6e5fe05cda99859ab6160a22a250cb31 not found: ID does not exist"
Sep 17 21:19:13.356531 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356511    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kserve-provision-location\") pod \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") "
Sep 17 21:19:13.356612 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356546    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc4zl\" (UniqueName: \"kubernetes.io/projected/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-api-access-hc4zl\") pod \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") "
Sep 17 21:19:13.356612 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356578    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-cabundle-cert\") pod \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") "
Sep 17 21:19:13.356612 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356607    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-rbac-proxy-sar-config\") pod \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") "
Sep 17 21:19:13.356765 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356626    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-proxy-tls\") pod \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\" (UID: \"ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b\") "
Sep 17 21:19:13.356912 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356850    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" (UID: "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:19:13.356975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356957    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" (UID: "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:19:13.357014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.356987    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" (UID: "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:19:13.358404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.358382    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-api-access-hc4zl" (OuterVolumeSpecName: "kube-api-access-hc4zl") pod "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" (UID: "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b"). InnerVolumeSpecName "kube-api-access-hc4zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:19:13.358573 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.358549    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" (UID: "ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:19:13.457502 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.457479    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:19:13.457502 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.457500    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc4zl\" (UniqueName: \"kubernetes.io/projected/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-api-access-hc4zl\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:19:13.457695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.457510    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:19:13.457695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.457520    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:19:13.457695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.457528    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:19:13.645203 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.645176    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"]
Sep 17 21:19:13.648934 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:13.648909    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-hhg8w"]
Sep 17 21:19:14.670767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:14.670726    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" path="/var/lib/kubelet/pods/ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b/volumes"
Sep 17 21:19:16.336643 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:16.336614    2586 generic.go:356] "Generic (PLEG): container finished" podID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerID="b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574" exitCode=0
Sep 17 21:19:16.337015 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:16.336689    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerDied","Data":"b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574"}
Sep 17 21:19:17.342046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:17.342010    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerStarted","Data":"afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7"}
Sep 17 21:19:17.342479 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:17.342068    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerStarted","Data":"7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255"}
Sep 17 21:19:17.342479 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:17.342264    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:17.389731 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:17.389686    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podStartSLOduration=2.599166173 podStartE2EDuration="6.389670065s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.790503892s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:19:17.389336612 +0000 UTC m=+1772.923276422" watchObservedRunningTime="2026-09-17 21:19:17.389670065 +0000 UTC m=+1772.923609873"
Sep 17 21:19:18.345924 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:18.345894    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:18.347202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:18.347174    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:19:19.349308 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:19.349263    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:19:21.294634 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:21.294592    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:19:22.299628 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:22.299590    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:19:24.354334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:24.354305    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:19:24.354948 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:24.354909    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:19:31.294909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:31.294831    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:19:32.299660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:32.299621    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:19:34.355075 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:34.355033    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:19:41.294751 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:41.294714    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:19:42.299782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:42.299745    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:19:44.355135 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:44.355092    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:19:46.656613 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:46.656586    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:19:46.658439 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:46.658420    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:19:51.295474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:51.295438    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:19:52.299678 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:52.299642    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:19:54.354869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:19:54.354826    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:20:01.295186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:01.295149    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:20:02.299844 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:02.299805    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:20:04.355438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:04.355399    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:20:11.295244 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:11.295202    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:20:12.300045 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:12.300007    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:20:14.355584 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:14.355541    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:20:17.666662 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:17.666624    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 21:20:22.299542 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:22.299508    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:20:22.424890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:22.424860    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"]
Sep 17 21:20:22.425093 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:22.425053    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container" containerID="cri-o://72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349" gracePeriod=30
Sep 17 21:20:22.425218 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:22.425128    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kube-rbac-proxy" containerID="cri-o://b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86" gracePeriod=30
Sep 17 21:20:22.425974 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:22.425954    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:20:22.588562 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:22.588487    2586 generic.go:356] "Generic (PLEG): container finished" podID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerID="b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86" exitCode=2
Sep 17 21:20:22.588562 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:22.588536    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerDied","Data":"b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86"}
Sep 17 21:20:24.355131 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:24.355093    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:20:26.290409 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.290351    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 21:20:26.856425 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.856402    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:20:26.996423 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996326    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-cabundle-cert\") pod \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") "
Sep 17 21:20:26.996577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996446    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kserve-provision-location\") pod \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") "
Sep 17 21:20:26.996577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996479    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5164c1b-3b21-476e-9c92-8f5fda9ad063-proxy-tls\") pod \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") "
Sep 17 21:20:26.996577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996502    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-api-access-jhwxz\") pod \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") "
Sep 17 21:20:26.996577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996530    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-rbac-proxy-sar-config\") pod \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\" (UID: \"c5164c1b-3b21-476e-9c92-8f5fda9ad063\") "
Sep 17 21:20:26.996770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996720    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5164c1b-3b21-476e-9c92-8f5fda9ad063" (UID: "c5164c1b-3b21-476e-9c92-8f5fda9ad063"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:26.996770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996734    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5164c1b-3b21-476e-9c92-8f5fda9ad063" (UID: "c5164c1b-3b21-476e-9c92-8f5fda9ad063"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:26.996921 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.996893    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5164c1b-3b21-476e-9c92-8f5fda9ad063" (UID: "c5164c1b-3b21-476e-9c92-8f5fda9ad063"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:26.998504 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.998480    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-api-access-jhwxz" (OuterVolumeSpecName: "kube-api-access-jhwxz") pod "c5164c1b-3b21-476e-9c92-8f5fda9ad063" (UID: "c5164c1b-3b21-476e-9c92-8f5fda9ad063"). InnerVolumeSpecName "kube-api-access-jhwxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:20:26.998568 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:26.998543    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5164c1b-3b21-476e-9c92-8f5fda9ad063-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5164c1b-3b21-476e-9c92-8f5fda9ad063" (UID: "c5164c1b-3b21-476e-9c92-8f5fda9ad063"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:20:27.097567 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.097544    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5164c1b-3b21-476e-9c92-8f5fda9ad063-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:27.097567 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.097564    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-api-access-jhwxz\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:27.097567 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.097573    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:27.097759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.097582    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5164c1b-3b21-476e-9c92-8f5fda9ad063-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:27.097759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.097591    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5164c1b-3b21-476e-9c92-8f5fda9ad063-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:27.608989 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.608957    2586 generic.go:356] "Generic (PLEG): container finished" podID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerID="72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349" exitCode=0
Sep 17 21:20:27.609434 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.609009    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"
Sep 17 21:20:27.609434 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.609020    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerDied","Data":"72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349"}
Sep 17 21:20:27.609434 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.609066    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g" event={"ID":"c5164c1b-3b21-476e-9c92-8f5fda9ad063","Type":"ContainerDied","Data":"79ed3f3053b217c925f42a0c7d9abead7ed8cd3fa6fc7a992d8aeca7440116d1"}
Sep 17 21:20:27.609434 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.609091    2586 scope.go:117] "RemoveContainer" containerID="b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86"
Sep 17 21:20:27.617651 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.617632    2586 scope.go:117] "RemoveContainer" containerID="72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349"
Sep 17 21:20:27.624779 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.624763    2586 scope.go:117] "RemoveContainer" containerID="bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee"
Sep 17 21:20:27.632218 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.632198    2586 scope.go:117] "RemoveContainer" containerID="b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86"
Sep 17 21:20:27.632284 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.632218    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"]
Sep 17 21:20:27.632505 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:27.632485    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86\": container with ID starting with b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86 not found: ID does not exist" containerID="b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86"
Sep 17 21:20:27.632554 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.632513    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86"} err="failed to get container status \"b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86\": rpc error: code = NotFound desc = could not find container \"b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86\": container with ID starting with b44f2efcece772f2bf22815cd751c012f87700152d807badfb65f5ceb2a48b86 not found: ID does not exist"
Sep 17 21:20:27.632554 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.632529    2586 scope.go:117] "RemoveContainer" containerID="72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349"
Sep 17 21:20:27.632780 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:27.632763    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349\": container with ID starting with 72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349 not found: ID does not exist" containerID="72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349"
Sep 17 21:20:27.632847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.632789    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349"} err="failed to get container status \"72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349\": rpc error: code = NotFound desc = could not find container \"72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349\": container with ID starting with 72ef2c9a0f4fb048735e17c21e97b031452a602648c0cf9c34c5faff91b35349 not found: ID does not exist"
Sep 17 21:20:27.632847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.632810    2586 scope.go:117] "RemoveContainer" containerID="bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee"
Sep 17 21:20:27.633099 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:27.633084    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee\": container with ID starting with bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee not found: ID does not exist" containerID="bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee"
Sep 17 21:20:27.633139 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.633104    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee"} err="failed to get container status \"bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee\": rpc error: code = NotFound desc = could not find container \"bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee\": container with ID starting with bb003372914fde0458896f840998efc25eaef66cd8a89d298c2f866df70f59ee not found: ID does not exist"
Sep 17 21:20:27.640450 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:27.640429    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-8cl4g"]
Sep 17 21:20:28.670646 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:28.670612    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" path="/var/lib/kubelet/pods/c5164c1b-3b21-476e-9c92-8f5fda9ad063/volumes"
Sep 17 21:20:29.666911 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:29.666871    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 21:20:39.667414 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:39.667382    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:20:41.912736 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:41.912702    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"]
Sep 17 21:20:41.913091 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:41.912941    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container" containerID="cri-o://7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255" gracePeriod=30
Sep 17 21:20:41.913091 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:41.912981    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kube-rbac-proxy" containerID="cri-o://afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7" gracePeriod=30
Sep 17 21:20:42.671969 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:42.671939    2586 generic.go:356] "Generic (PLEG): container finished" podID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerID="afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7" exitCode=2
Sep 17 21:20:42.672140 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:42.672001    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerDied","Data":"afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7"}
Sep 17 21:20:44.350261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:44.350222    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 21:20:46.466626 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.466601    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:20:46.633592 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.633557    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-cabundle-cert\") pod \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") "
Sep 17 21:20:46.633767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.633619    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pt9w\" (UniqueName: \"kubernetes.io/projected/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-api-access-7pt9w\") pod \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") "
Sep 17 21:20:46.633767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.633637    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-rbac-proxy-sar-config\") pod \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") "
Sep 17 21:20:46.633767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.633674    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kserve-provision-location\") pod \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") "
Sep 17 21:20:46.633767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.633700    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls\") pod \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\" (UID: \"7c851cf5-ddaa-4fea-af35-958c7fc9f623\") "
Sep 17 21:20:46.634040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.634015    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c851cf5-ddaa-4fea-af35-958c7fc9f623" (UID: "7c851cf5-ddaa-4fea-af35-958c7fc9f623"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:46.634147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.634041    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7c851cf5-ddaa-4fea-af35-958c7fc9f623" (UID: "7c851cf5-ddaa-4fea-af35-958c7fc9f623"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:46.634147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.634105    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c851cf5-ddaa-4fea-af35-958c7fc9f623" (UID: "7c851cf5-ddaa-4fea-af35-958c7fc9f623"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:46.635738 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.635720    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-api-access-7pt9w" (OuterVolumeSpecName: "kube-api-access-7pt9w") pod "7c851cf5-ddaa-4fea-af35-958c7fc9f623" (UID: "7c851cf5-ddaa-4fea-af35-958c7fc9f623"). InnerVolumeSpecName "kube-api-access-7pt9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:20:46.635913 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.635892    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c851cf5-ddaa-4fea-af35-958c7fc9f623" (UID: "7c851cf5-ddaa-4fea-af35-958c7fc9f623"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:20:46.690365 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.690329    2586 generic.go:356] "Generic (PLEG): container finished" podID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerID="7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255" exitCode=0
Sep 17 21:20:46.690461 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.690392    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"
Sep 17 21:20:46.690461 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.690412    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerDied","Data":"7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255"}
Sep 17 21:20:46.690536 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.690465    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7" event={"ID":"7c851cf5-ddaa-4fea-af35-958c7fc9f623","Type":"ContainerDied","Data":"718df4dabe52942e8335accd0825f9ad6f34a492423febcb50de03ed2478a090"}
Sep 17 21:20:46.690536 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.690494    2586 scope.go:117] "RemoveContainer" containerID="afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7"
Sep 17 21:20:46.697895 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.697875    2586 scope.go:117] "RemoveContainer" containerID="7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255"
Sep 17 21:20:46.704738 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.704721    2586 scope.go:117] "RemoveContainer" containerID="b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574"
Sep 17 21:20:46.711340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.711324    2586 scope.go:117] "RemoveContainer" containerID="afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7"
Sep 17 21:20:46.711609 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:46.711583    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7\": container with ID starting with afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7 not found: ID does not exist" containerID="afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7"
Sep 17 21:20:46.711689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.711609    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7"} err="failed to get container status \"afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7\": rpc error: code = NotFound desc = could not find container \"afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7\": container with ID starting with afd8584a474a2629e46967bbc934585a4e4747b511db738209dc6cabd8146fa7 not found: ID does not exist"
Sep 17 21:20:46.711689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.711630    2586 scope.go:117] "RemoveContainer" containerID="7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255"
Sep 17 21:20:46.711848 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:46.711832    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255\": container with ID starting with 7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255 not found: ID does not exist" containerID="7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255"
Sep 17 21:20:46.711890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.711851    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255"} err="failed to get container status \"7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255\": rpc error: code = NotFound desc = could not find container \"7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255\": container with ID starting with 7dc4b0213811d0343ba13d772878b908aa748c1fef2346fff5de220e2884b255 not found: ID does not exist"
Sep 17 21:20:46.711890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.711863    2586 scope.go:117] "RemoveContainer" containerID="b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574"
Sep 17 21:20:46.712108 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:46.712094    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574\": container with ID starting with b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574 not found: ID does not exist" containerID="b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574"
Sep 17 21:20:46.712159 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.712110    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574"} err="failed to get container status \"b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574\": rpc error: code = NotFound desc = could not find container \"b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574\": container with ID starting with b459b89de10ec7964212e6a06abef84fd628ef675b1c841b98bf458f507e4574 not found: ID does not exist"
Sep 17 21:20:46.717491 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.717467    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"]
Sep 17 21:20:46.720533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.720516    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-hq7g7"]
Sep 17 21:20:46.734787 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.734770    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:46.734858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.734788    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c851cf5-ddaa-4fea-af35-958c7fc9f623-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:46.734858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.734798    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:46.734858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.734806    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pt9w\" (UniqueName: \"kubernetes.io/projected/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-api-access-7pt9w\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:46.734858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:46.734815    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c851cf5-ddaa-4fea-af35-958c7fc9f623-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:47.288454 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288420    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"]
Sep 17 21:20:47.288742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288725    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kube-rbac-proxy"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288745    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kube-rbac-proxy"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288761    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kserve-container"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288769    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kserve-container"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288779    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kube-rbac-proxy"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288787    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kube-rbac-proxy"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288796    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="storage-initializer"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288805    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="storage-initializer"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288817    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kube-rbac-proxy"
Sep 17 21:20:47.288824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288826    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kube-rbac-proxy"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288843    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288851    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288861    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="storage-initializer"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288869    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="storage-initializer"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288880    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="storage-initializer"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288889    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="storage-initializer"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288906    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288915    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288973    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kserve-container"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288984    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kserve-container"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.288997    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5164c1b-3b21-476e-9c92-8f5fda9ad063" containerName="kube-rbac-proxy"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.289006    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kserve-container"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.289014    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" containerName="kube-rbac-proxy"
Sep 17 21:20:47.289316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.289023    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3aa038-6f0d-43fb-b8fb-156cf9fdd44b" containerName="kube-rbac-proxy"
Sep 17 21:20:47.293423 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.293403    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.299703 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.299680    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 21:20:47.301057 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.301035    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 21:20:47.318875 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.318856    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"]
Sep 17 21:20:47.440328 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.440299    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.440469 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.440332    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rgg\" (UniqueName: \"kubernetes.io/projected/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-api-access-46rgg\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.440469 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.440351    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.440550 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.440463    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.440550 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.440494    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541151 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541085    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541151 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541122    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541616 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541177    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541616 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rgg\" (UniqueName: \"kubernetes.io/projected/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-api-access-46rgg\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541616 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541234    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541616 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541549    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541837 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541794    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.541967 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.541946    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.543617 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.543599    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.550647 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.550629    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rgg\" (UniqueName: \"kubernetes.io/projected/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-api-access-46rgg\") pod \"isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.603552 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.603528    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:47.876474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:47.876453    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"]
Sep 17 21:20:47.878626 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:20:47.878599    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f48c5b_3b9a_46ef_a5b3_d0931a0e39f5.slice/crio-ed806b07f7cdfcccd14c0987a1055aa97a2304aa6fd7fed434f0dedcc9310cdc WatchSource:0}: Error finding container ed806b07f7cdfcccd14c0987a1055aa97a2304aa6fd7fed434f0dedcc9310cdc: Status 404 returned error can't find the container with id ed806b07f7cdfcccd14c0987a1055aa97a2304aa6fd7fed434f0dedcc9310cdc
Sep 17 21:20:48.670927 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:48.670889    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c851cf5-ddaa-4fea-af35-958c7fc9f623" path="/var/lib/kubelet/pods/7c851cf5-ddaa-4fea-af35-958c7fc9f623/volumes"
Sep 17 21:20:48.700125 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:48.700095    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerStarted","Data":"7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a"}
Sep 17 21:20:48.700277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:48.700152    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerStarted","Data":"ed806b07f7cdfcccd14c0987a1055aa97a2304aa6fd7fed434f0dedcc9310cdc"}
Sep 17 21:20:49.573571 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:49.573540    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"]
Sep 17 21:20:49.573799 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:49.573773    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" containerID="cri-o://c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2" gracePeriod=30
Sep 17 21:20:49.573894 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:49.573833    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kube-rbac-proxy" containerID="cri-o://cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9" gracePeriod=30
Sep 17 21:20:49.705166 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:49.705140    2586 generic.go:356] "Generic (PLEG): container finished" podID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerID="cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9" exitCode=2
Sep 17 21:20:49.705556 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:49.705213    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerDied","Data":"cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9"}
Sep 17 21:20:52.294806 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:52.294720    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 21:20:52.299088 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:52.299062    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 21:20:52.717224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:52.717190    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerID="7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a" exitCode=0
Sep 17 21:20:52.717424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:52.717240    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerDied","Data":"7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a"}
Sep 17 21:20:53.723886 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:53.723855    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerStarted","Data":"5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e"}
Sep 17 21:20:53.724249 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:53.723916    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerStarted","Data":"54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c"}
Sep 17 21:20:53.724249 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:53.724190    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:53.724379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:53.724340    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:53.725757 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:53.725729    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:20:53.744774 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:53.744733    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podStartSLOduration=2.78904201 podStartE2EDuration="6.744718774s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.955676764s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:20:53.743026905 +0000 UTC m=+1869.276966715" watchObservedRunningTime="2026-09-17 21:20:53.744718774 +0000 UTC m=+1869.278658583"
Sep 17 21:20:54.109044 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.109020    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:20:54.186570 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.186541    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-proxy-tls\") pod \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") "
Sep 17 21:20:54.186719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.186581    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-rbac-proxy-sar-config\") pod \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") "
Sep 17 21:20:54.186719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.186618    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kserve-provision-location\") pod \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") "
Sep 17 21:20:54.186719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.186650    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcsbf\" (UniqueName: \"kubernetes.io/projected/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-api-access-dcsbf\") pod \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") "
Sep 17 21:20:54.186719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.186674    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-cabundle-cert\") pod \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\" (UID: \"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a\") "
Sep 17 21:20:54.187000 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.186973    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" (UID: "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:20:54.187060 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.187018    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" (UID: "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:54.187060 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.187015    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" (UID: "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:20:54.188582 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.188558    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" (UID: "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:20:54.188771 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.188756    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-api-access-dcsbf" (OuterVolumeSpecName: "kube-api-access-dcsbf") pod "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" (UID: "0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a"). InnerVolumeSpecName "kube-api-access-dcsbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:20:54.287404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.287320    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.287404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.287346    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcsbf\" (UniqueName: \"kubernetes.io/projected/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-api-access-dcsbf\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.287404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.287377    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.287404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.287389    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.287404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.287402    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:20:54.729553 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.729522    2586 generic.go:356] "Generic (PLEG): container finished" podID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerID="c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2" exitCode=0
Sep 17 21:20:54.729988 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.729602    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"
Sep 17 21:20:54.729988 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.729615    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerDied","Data":"c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2"}
Sep 17 21:20:54.729988 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.729692    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw" event={"ID":"0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a","Type":"ContainerDied","Data":"9f1cd1e0d4eae27148718216046d60db137c2226861c2d00438d8e3ba036ebbb"}
Sep 17 21:20:54.729988 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.729733    2586 scope.go:117] "RemoveContainer" containerID="cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9"
Sep 17 21:20:54.730474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.730448    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:20:54.738303 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.738284    2586 scope.go:117] "RemoveContainer" containerID="c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2"
Sep 17 21:20:54.747448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.746118    2586 scope.go:117] "RemoveContainer" containerID="6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7"
Sep 17 21:20:54.755229 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.755209    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"]
Sep 17 21:20:54.755427 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.755348    2586 scope.go:117] "RemoveContainer" containerID="cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9"
Sep 17 21:20:54.755680 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:54.755659    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9\": container with ID starting with cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9 not found: ID does not exist" containerID="cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9"
Sep 17 21:20:54.755727 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.755688    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9"} err="failed to get container status \"cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9\": rpc error: code = NotFound desc = could not find container \"cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9\": container with ID starting with cd21c92cdfaed04afb6f9e2779c1f9e7983a2a859418670271500591180b8ec9 not found: ID does not exist"
Sep 17 21:20:54.755727 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.755704    2586 scope.go:117] "RemoveContainer" containerID="c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2"
Sep 17 21:20:54.756083 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:54.756062    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2\": container with ID starting with c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2 not found: ID does not exist" containerID="c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2"
Sep 17 21:20:54.756132 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.756094    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2"} err="failed to get container status \"c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2\": rpc error: code = NotFound desc = could not find container \"c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2\": container with ID starting with c3c25bfe4f6afd6847d09beb0dd0d496e68a2e0c454e65b904f4e6f58fc7a4a2 not found: ID does not exist"
Sep 17 21:20:54.756132 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.756116    2586 scope.go:117] "RemoveContainer" containerID="6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7"
Sep 17 21:20:54.756412 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:20:54.756394    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7\": container with ID starting with 6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7 not found: ID does not exist" containerID="6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7"
Sep 17 21:20:54.756470 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.756419    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7"} err="failed to get container status \"6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7\": rpc error: code = NotFound desc = could not find container \"6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7\": container with ID starting with 6a507f6fbaacced837c9c5199233e34048ebe41c041899b668acbdfc8be624a7 not found: ID does not exist"
Sep 17 21:20:54.760579 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.760559    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-bfcdb48dd-hksnw"]
Sep 17 21:20:54.771145 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771126    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"]
Sep 17 21:20:54.771364 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771344    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container"
Sep 17 21:20:54.771405 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771375    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container"
Sep 17 21:20:54.771405 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771390    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="storage-initializer"
Sep 17 21:20:54.771405 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771396    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="storage-initializer"
Sep 17 21:20:54.771405 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771405    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kube-rbac-proxy"
Sep 17 21:20:54.771533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771411    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kube-rbac-proxy"
Sep 17 21:20:54.771533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771458    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kube-rbac-proxy"
Sep 17 21:20:54.771533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.771467    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" containerName="kserve-container"
Sep 17 21:20:54.775600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.775585    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.778254 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.778238    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 21:20:54.778254 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.778244    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 21:20:54.790784 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.790767    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"]
Sep 17 21:20:54.815762 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.815739    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"]
Sep 17 21:20:54.821635 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.821621    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:54.824095 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.824077    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 21:20:54.830637 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.830617    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"]
Sep 17 21:20:54.891162 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.891134    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.891278 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.891170    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65c3cc9-2338-43df-a900-a257b4d6f2ff-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.891278 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.891209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.891419 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.891392    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.891481 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.891463    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24x4\" (UniqueName: \"kubernetes.io/projected/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-api-access-l24x4\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.992559 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992498    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f822d8d1-135d-4577-9aff-14e47b3640a6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:54.992559 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.992704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992577    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24x4\" (UniqueName: \"kubernetes.io/projected/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-api-access-l24x4\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.992704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992594    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:54.992704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992621    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.992704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992642    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79v26\" (UniqueName: \"kubernetes.io/projected/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-api-access-79v26\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:54.992704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992684    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:54.992931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992761    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65c3cc9-2338-43df-a900-a257b4d6f2ff-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.992931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.992931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.992809    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f822d8d1-135d-4577-9aff-14e47b3640a6-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:54.993168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.993135    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.993262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.993224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.993262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.993234    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:54.995065 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:54.995045    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65c3cc9-2338-43df-a900-a257b4d6f2ff-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:55.001165 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.001146    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24x4\" (UniqueName: \"kubernetes.io/projected/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-api-access-l24x4\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:55.084946 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.084918    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:20:55.094093 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.094178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79v26\" (UniqueName: \"kubernetes.io/projected/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-api-access-79v26\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.094178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094145    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.094255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094185    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f822d8d1-135d-4577-9aff-14e47b3640a6-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.094255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094229    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f822d8d1-135d-4577-9aff-14e47b3640a6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.094769 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094743    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f822d8d1-135d-4577-9aff-14e47b3640a6-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.094843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094784    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.094899 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.094858    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.096628 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.096610    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f822d8d1-135d-4577-9aff-14e47b3640a6-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.119669 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.119640    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79v26\" (UniqueName: \"kubernetes.io/projected/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-api-access-79v26\") pod \"isvc-canary-promote-predictor-78b4c78fb9-nznmv\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.131311 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.131283    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:20:55.355935 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.355895    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"]
Sep 17 21:20:55.357847 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:20:55.357826    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65c3cc9_2338_43df_a900_a257b4d6f2ff.slice/crio-61258dd8358e7a6fa276d59a94cf82b553954d291a0ce643304bce61170055b1 WatchSource:0}: Error finding container 61258dd8358e7a6fa276d59a94cf82b553954d291a0ce643304bce61170055b1: Status 404 returned error can't find the container with id 61258dd8358e7a6fa276d59a94cf82b553954d291a0ce643304bce61170055b1
Sep 17 21:20:55.405090 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.405063    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"]
Sep 17 21:20:55.406713 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:20:55.406690    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf822d8d1_135d_4577_9aff_14e47b3640a6.slice/crio-ddd563d2acd5b6a0bbad07c606305e4b413a985801d64a3195881e2a1dbd0762 WatchSource:0}: Error finding container ddd563d2acd5b6a0bbad07c606305e4b413a985801d64a3195881e2a1dbd0762: Status 404 returned error can't find the container with id ddd563d2acd5b6a0bbad07c606305e4b413a985801d64a3195881e2a1dbd0762
Sep 17 21:20:55.737542 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.737448    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerStarted","Data":"5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7"}
Sep 17 21:20:55.738115 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.737618    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerStarted","Data":"ddd563d2acd5b6a0bbad07c606305e4b413a985801d64a3195881e2a1dbd0762"}
Sep 17 21:20:55.740076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.740051    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerStarted","Data":"36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39"}
Sep 17 21:20:55.740236 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:55.740105    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerStarted","Data":"61258dd8358e7a6fa276d59a94cf82b553954d291a0ce643304bce61170055b1"}
Sep 17 21:20:56.671766 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:56.671730    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a" path="/var/lib/kubelet/pods/0120be8b-5ad6-4d5c-a0ab-5eed89aa8f7a/volumes"
Sep 17 21:20:59.733728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:59.733655    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:20:59.734221 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:59.734192    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:20:59.758062 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:59.758037    2586 generic.go:356] "Generic (PLEG): container finished" podID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerID="5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7" exitCode=0
Sep 17 21:20:59.758220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:59.758108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerDied","Data":"5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7"}
Sep 17 21:20:59.759691 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:59.759668    2586 generic.go:356] "Generic (PLEG): container finished" podID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerID="36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39" exitCode=0
Sep 17 21:20:59.759778 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:20:59.759707    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerDied","Data":"36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39"}
Sep 17 21:21:00.765985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.765949    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerStarted","Data":"189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d"}
Sep 17 21:21:00.766446 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.766014    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerStarted","Data":"a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580"}
Sep 17 21:21:00.766446 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.766301    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:21:00.766446 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.766375    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:21:00.767812 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.767787    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:00.768255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.768237    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerStarted","Data":"74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8"}
Sep 17 21:21:00.768322 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.768280    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerStarted","Data":"43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882"}
Sep 17 21:21:00.768526 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.768511    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:21:00.768569 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.768550    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:21:00.769498 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.769473    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:00.786493 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.786451    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podStartSLOduration=2.798120063 podStartE2EDuration="6.786439569s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.988319506s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:00.786228361 +0000 UTC m=+1876.320168171" watchObservedRunningTime="2026-09-17 21:21:00.786439569 +0000 UTC m=+1876.320379378"
Sep 17 21:21:00.805890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:00.805853    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podStartSLOduration=2.810032703 podStartE2EDuration="6.805843137s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.995810434s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:21:00.804845255 +0000 UTC m=+1876.338785065" watchObservedRunningTime="2026-09-17 21:21:00.805843137 +0000 UTC m=+1876.339782945"
Sep 17 21:21:01.771961 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:01.771925    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:01.772383 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:01.771958    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:06.776427 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:06.776392    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:21:06.776917 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:06.776808    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:21:06.776917 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:06.776898    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:06.777248 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:06.777225    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:09.735031 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:09.734995    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:21:16.777315 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:16.777278    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:16.777717 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:16.777279    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:19.734415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:19.734378    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:21:26.777238 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:26.777201    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:26.779616 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:26.777204    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:29.734225 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:29.734187    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:21:36.777446 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:36.777404    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:36.777830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:36.777404    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:39.734824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:39.734779    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:21:46.776808 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:46.776769    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:46.777179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:46.777152    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:49.734522 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:49.734478    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:21:56.777425 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:56.777383    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:21:56.777824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:56.777383    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:21:59.735097 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:21:59.735059    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:22:06.776871 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:06.776833    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:22:06.777256 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:06.777232    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:22:08.667290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:08.667240    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 21:22:09.734891 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:09.734862    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:22:16.777534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:16.777502    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:22:16.911415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:16.911378    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"]
Sep 17 21:22:16.911660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:16.911627    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container" containerID="cri-o://a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580" gracePeriod=30
Sep 17 21:22:16.911797 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:16.911695    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kube-rbac-proxy" containerID="cri-o://189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d" gracePeriod=30
Sep 17 21:22:16.912550 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:16.912531    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:22:17.064909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:17.064880    2586 generic.go:356] "Generic (PLEG): container finished" podID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerID="189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d" exitCode=2
Sep 17 21:22:17.065063 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:17.064917    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerDied","Data":"189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d"}
Sep 17 21:22:17.959076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:17.959041    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"]
Sep 17 21:22:17.959461 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:17.959269    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" containerID="cri-o://54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c" gracePeriod=30
Sep 17 21:22:17.959461 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:17.959302    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kube-rbac-proxy" containerID="cri-o://5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e" gracePeriod=30
Sep 17 21:22:19.074304 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:19.074269    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerID="5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e" exitCode=2
Sep 17 21:22:19.074684 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:19.074334    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerDied","Data":"5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e"}
Sep 17 21:22:19.730865 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:19.730823    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 21:22:19.734547 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:19.734522    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 21:22:21.445422 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.445398    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:22:21.512079 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512005    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79v26\" (UniqueName: \"kubernetes.io/projected/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-api-access-79v26\") pod \"f822d8d1-135d-4577-9aff-14e47b3640a6\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") "
Sep 17 21:22:21.512079 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512048    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-rbac-proxy-sar-config\") pod \"f822d8d1-135d-4577-9aff-14e47b3640a6\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") "
Sep 17 21:22:21.512277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512082    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f822d8d1-135d-4577-9aff-14e47b3640a6-proxy-tls\") pod \"f822d8d1-135d-4577-9aff-14e47b3640a6\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") "
Sep 17 21:22:21.512277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512112    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-cabundle-cert\") pod \"f822d8d1-135d-4577-9aff-14e47b3640a6\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") "
Sep 17 21:22:21.512277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512144    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f822d8d1-135d-4577-9aff-14e47b3640a6-kserve-provision-location\") pod \"f822d8d1-135d-4577-9aff-14e47b3640a6\" (UID: \"f822d8d1-135d-4577-9aff-14e47b3640a6\") "
Sep 17 21:22:21.512601 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512569    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f822d8d1-135d-4577-9aff-14e47b3640a6" (UID: "f822d8d1-135d-4577-9aff-14e47b3640a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:21.512760 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512639    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f822d8d1-135d-4577-9aff-14e47b3640a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f822d8d1-135d-4577-9aff-14e47b3640a6" (UID: "f822d8d1-135d-4577-9aff-14e47b3640a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:22:21.512853 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.512678    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f822d8d1-135d-4577-9aff-14e47b3640a6" (UID: "f822d8d1-135d-4577-9aff-14e47b3640a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:21.514323 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.514270    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f822d8d1-135d-4577-9aff-14e47b3640a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f822d8d1-135d-4577-9aff-14e47b3640a6" (UID: "f822d8d1-135d-4577-9aff-14e47b3640a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:22:21.515369 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.515326    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-api-access-79v26" (OuterVolumeSpecName: "kube-api-access-79v26") pod "f822d8d1-135d-4577-9aff-14e47b3640a6" (UID: "f822d8d1-135d-4577-9aff-14e47b3640a6"). InnerVolumeSpecName "kube-api-access-79v26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:22:21.613608 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.613582    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79v26\" (UniqueName: \"kubernetes.io/projected/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-api-access-79v26\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:21.613608 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.613605    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:21.613759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.613617    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f822d8d1-135d-4577-9aff-14e47b3640a6-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:21.613759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.613626    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f822d8d1-135d-4577-9aff-14e47b3640a6-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:21.613759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:21.613634    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f822d8d1-135d-4577-9aff-14e47b3640a6-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:22.087018 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.086986    2586 generic.go:356] "Generic (PLEG): container finished" podID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerID="a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580" exitCode=0
Sep 17 21:22:22.087221 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.087033    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerDied","Data":"a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580"}
Sep 17 21:22:22.087221 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.087049    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"
Sep 17 21:22:22.087221 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.087079    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv" event={"ID":"f822d8d1-135d-4577-9aff-14e47b3640a6","Type":"ContainerDied","Data":"ddd563d2acd5b6a0bbad07c606305e4b413a985801d64a3195881e2a1dbd0762"}
Sep 17 21:22:22.087221 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.087105    2586 scope.go:117] "RemoveContainer" containerID="189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d"
Sep 17 21:22:22.098394 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.098376    2586 scope.go:117] "RemoveContainer" containerID="a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580"
Sep 17 21:22:22.105527 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.105511    2586 scope.go:117] "RemoveContainer" containerID="5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7"
Sep 17 21:22:22.111517 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.111491    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"]
Sep 17 21:22:22.113191 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.113177    2586 scope.go:117] "RemoveContainer" containerID="189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d"
Sep 17 21:22:22.113528 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:22.113502    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d\": container with ID starting with 189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d not found: ID does not exist" containerID="189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d"
Sep 17 21:22:22.113620 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.113539    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d"} err="failed to get container status \"189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d\": rpc error: code = NotFound desc = could not find container \"189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d\": container with ID starting with 189c6c4ddb112dd3c568b1ec78940823f69d3a44f01bd5995d11999a54bfac2d not found: ID does not exist"
Sep 17 21:22:22.113620 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.113563    2586 scope.go:117] "RemoveContainer" containerID="a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580"
Sep 17 21:22:22.114565 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:22.114544    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580\": container with ID starting with a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580 not found: ID does not exist" containerID="a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580"
Sep 17 21:22:22.114654 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.114570    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580"} err="failed to get container status \"a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580\": rpc error: code = NotFound desc = could not find container \"a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580\": container with ID starting with a012483efa886be8268e6666bb6ce9866ea7b303fbf6fce3fdb14ecca35ff580 not found: ID does not exist"
Sep 17 21:22:22.114654 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.114586    2586 scope.go:117] "RemoveContainer" containerID="5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7"
Sep 17 21:22:22.114854 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:22.114836    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7\": container with ID starting with 5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7 not found: ID does not exist" containerID="5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7"
Sep 17 21:22:22.114911 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.114873    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7"} err="failed to get container status \"5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7\": rpc error: code = NotFound desc = could not find container \"5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7\": container with ID starting with 5d0f69e47aa7976404db3abfc435c43b9e5cfede1b7ad39d24bc19a2934091c7 not found: ID does not exist"
Sep 17 21:22:22.116403 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.116382    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-nznmv"]
Sep 17 21:22:22.670522 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.670495    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" path="/var/lib/kubelet/pods/f822d8d1-135d-4577-9aff-14e47b3640a6/volumes"
Sep 17 21:22:22.802859 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.802838    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:22:22.923280 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923208    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-rbac-proxy-sar-config\") pod \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") "
Sep 17 21:22:22.923280 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923260    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-proxy-tls\") pod \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") "
Sep 17 21:22:22.923280 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923280    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-cabundle-cert\") pod \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") "
Sep 17 21:22:22.923572 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923404    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kserve-provision-location\") pod \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") "
Sep 17 21:22:22.923572 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923443    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46rgg\" (UniqueName: \"kubernetes.io/projected/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-api-access-46rgg\") pod \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\" (UID: \"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5\") "
Sep 17 21:22:22.923676 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923627    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" (UID: "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:22.923718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923689    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" (UID: "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:22:22.923752 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.923694    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" (UID: "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:22:22.925496 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.925474    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" (UID: "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:22:22.925599 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:22.925581    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-api-access-46rgg" (OuterVolumeSpecName: "kube-api-access-46rgg") pod "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" (UID: "66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5"). InnerVolumeSpecName "kube-api-access-46rgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:22:23.024010 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.023977    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:23.024010 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.024002    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:23.024010 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.024011    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:23.024231 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.024020    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46rgg\" (UniqueName: \"kubernetes.io/projected/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-api-access-46rgg\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:23.024231 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.024030    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:22:23.057793 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.057761    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"]
Sep 17 21:22:23.058026 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058015    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058028    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058035    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="storage-initializer"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058040    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="storage-initializer"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058051    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058056    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058065    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kube-rbac-proxy"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058070    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kube-rbac-proxy"
Sep 17 21:22:23.058076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058076    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kube-rbac-proxy"
Sep 17 21:22:23.058379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058083    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kube-rbac-proxy"
Sep 17 21:22:23.058379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058093    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="storage-initializer"
Sep 17 21:22:23.058379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058098    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="storage-initializer"
Sep 17 21:22:23.058379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058137    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kube-rbac-proxy"
Sep 17 21:22:23.058379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058146    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kube-rbac-proxy"
Sep 17 21:22:23.058379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058153    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerName="kserve-container"
Sep 17 21:22:23.058379 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.058159    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f822d8d1-135d-4577-9aff-14e47b3640a6" containerName="kserve-container"
Sep 17 21:22:23.061240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.061224    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.063405 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.063381    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 21:22:23.063688 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.063673    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 21:22:23.072479 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.072453    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"]
Sep 17 21:22:23.093072 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.093046    2586 generic.go:356] "Generic (PLEG): container finished" podID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" containerID="54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c" exitCode=0
Sep 17 21:22:23.093193 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.093108    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"
Sep 17 21:22:23.093193 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.093132    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerDied","Data":"54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c"}
Sep 17 21:22:23.093295 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.093223    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s" event={"ID":"66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5","Type":"ContainerDied","Data":"ed806b07f7cdfcccd14c0987a1055aa97a2304aa6fd7fed434f0dedcc9310cdc"}
Sep 17 21:22:23.093295 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.093274    2586 scope.go:117] "RemoveContainer" containerID="5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e"
Sep 17 21:22:23.100877 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.100772    2586 scope.go:117] "RemoveContainer" containerID="54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c"
Sep 17 21:22:23.107840 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.107823    2586 scope.go:117] "RemoveContainer" containerID="7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a"
Sep 17 21:22:23.115016 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.114994    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"]
Sep 17 21:22:23.115635 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.115614    2586 scope.go:117] "RemoveContainer" containerID="5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e"
Sep 17 21:22:23.115920 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:23.115899    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e\": container with ID starting with 5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e not found: ID does not exist" containerID="5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e"
Sep 17 21:22:23.115979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.115926    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e"} err="failed to get container status \"5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e\": rpc error: code = NotFound desc = could not find container \"5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e\": container with ID starting with 5d6aec3ae6d18765a707ef771e05ececfd0258eb54d6323de8e5671f22fde81e not found: ID does not exist"
Sep 17 21:22:23.115979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.115943    2586 scope.go:117] "RemoveContainer" containerID="54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c"
Sep 17 21:22:23.116186 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:23.116168    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c\": container with ID starting with 54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c not found: ID does not exist" containerID="54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c"
Sep 17 21:22:23.116251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.116196    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c"} err="failed to get container status \"54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c\": rpc error: code = NotFound desc = could not find container \"54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c\": container with ID starting with 54b4b37efb61da8d538d883d4d7eadc7559cd030d5d3f781c1296c9fb8a20f0c not found: ID does not exist"
Sep 17 21:22:23.116251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.116218    2586 scope.go:117] "RemoveContainer" containerID="7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a"
Sep 17 21:22:23.116513 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:23.116480    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a\": container with ID starting with 7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a not found: ID does not exist" containerID="7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a"
Sep 17 21:22:23.116566 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.116519    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a"} err="failed to get container status \"7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a\": rpc error: code = NotFound desc = could not find container \"7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a\": container with ID starting with 7a6032f5f043ca55b1102fc3a0d48eac7714ce6d883d024007d6664e1f845c6a not found: ID does not exist"
Sep 17 21:22:23.118559 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.118540    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-749744fb9-4wb9s"]
Sep 17 21:22:23.124387 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.124365    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2368e074-19ab-4cc5-a772-b08c24b35660-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.124474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.124425    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.124474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.124449    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.124474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.124468    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjlzq\" (UniqueName: \"kubernetes.io/projected/2368e074-19ab-4cc5-a772-b08c24b35660-kube-api-access-wjlzq\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.124623 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.124482    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.225310 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.225228    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2368e074-19ab-4cc5-a772-b08c24b35660-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.225310 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.225267    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.225310 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.225284    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.225565 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:23.225425    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 21:22:23.225565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.225460    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjlzq\" (UniqueName: \"kubernetes.io/projected/2368e074-19ab-4cc5-a772-b08c24b35660-kube-api-access-wjlzq\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.225565 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:22:23.225505    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls podName:2368e074-19ab-4cc5-a772-b08c24b35660 nodeName:}" failed. No retries permitted until 2026-09-17 21:22:23.725486217 +0000 UTC m=+1959.259426008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-rlzq2" (UID: "2368e074-19ab-4cc5-a772-b08c24b35660") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 21:22:23.225565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.225522    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.225770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.225691    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2368e074-19ab-4cc5-a772-b08c24b35660-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.225977 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.225958    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.226092 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.226076    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.233662 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.233640    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjlzq\" (UniqueName: \"kubernetes.io/projected/2368e074-19ab-4cc5-a772-b08c24b35660-kube-api-access-wjlzq\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.730893 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.730861    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.733191 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.733173    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-rlzq2\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:23.970584 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:23.970558    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:24.240572 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:24.240504    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"]
Sep 17 21:22:24.243613 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:22:24.243587    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2368e074_19ab_4cc5_a772_b08c24b35660.slice/crio-8466c2e3b2885de98832402635f6e2c4cf00ec5b498f26b7454372d53eb429de WatchSource:0}: Error finding container 8466c2e3b2885de98832402635f6e2c4cf00ec5b498f26b7454372d53eb429de: Status 404 returned error can't find the container with id 8466c2e3b2885de98832402635f6e2c4cf00ec5b498f26b7454372d53eb429de
Sep 17 21:22:24.677567 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:24.677533    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5" path="/var/lib/kubelet/pods/66f48c5b-3b9a-46ef-a5b3-d0931a0e39f5/volumes"
Sep 17 21:22:25.105054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:25.104964    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerStarted","Data":"e3af4fea3b0ef9b8aa059e937b3b81abec80b17e0fbe2247781b7aa1612f8497"}
Sep 17 21:22:25.105054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:25.105027    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerStarted","Data":"8466c2e3b2885de98832402635f6e2c4cf00ec5b498f26b7454372d53eb429de"}
Sep 17 21:22:29.120590 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:29.120556    2586 generic.go:356] "Generic (PLEG): container finished" podID="2368e074-19ab-4cc5-a772-b08c24b35660" containerID="e3af4fea3b0ef9b8aa059e937b3b81abec80b17e0fbe2247781b7aa1612f8497" exitCode=0
Sep 17 21:22:29.120941 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:29.120628    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerDied","Data":"e3af4fea3b0ef9b8aa059e937b3b81abec80b17e0fbe2247781b7aa1612f8497"}
Sep 17 21:22:29.121764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:29.121750    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:22:33.141349 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:33.141314    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerStarted","Data":"da62a611fe2a09bc14813147592400d00529643b1fc84c525e2ed27446226596"}
Sep 17 21:22:33.141796 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:33.141396    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerStarted","Data":"ddc5e0cd6f88d8da634a314d91e4201e5864d5648dfd614d536be4110bc55c7e"}
Sep 17 21:22:33.141796 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:33.141683    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:33.141796 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:33.141743    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:33.143096 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:33.143073    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 21:22:33.159574 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:33.159532    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podStartSLOduration=1.925756495 podStartE2EDuration="10.159519434s" totalImagesPullingTime="3.595752087s" totalInitContainerRuntime="4.638010852s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:22:33.158754057 +0000 UTC m=+1968.692693865" watchObservedRunningTime="2026-09-17 21:22:33.159519434 +0000 UTC m=+1968.693459243"
Sep 17 21:22:34.146198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:34.146162    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 21:22:39.150472 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:39.150441    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:22:39.151020 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:39.150996    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 21:22:49.151827 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:22:49.151791    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:23:02.937811 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:02.937782    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"]
Sep 17 21:23:02.938270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:02.937990    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kserve-container" containerID="cri-o://ddc5e0cd6f88d8da634a314d91e4201e5864d5648dfd614d536be4110bc55c7e" gracePeriod=30
Sep 17 21:23:02.938270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:02.938052    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy" containerID="cri-o://da62a611fe2a09bc14813147592400d00529643b1fc84c525e2ed27446226596" gracePeriod=30
Sep 17 21:23:03.251797 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:03.251723    2586 generic.go:356] "Generic (PLEG): container finished" podID="2368e074-19ab-4cc5-a772-b08c24b35660" containerID="da62a611fe2a09bc14813147592400d00529643b1fc84c525e2ed27446226596" exitCode=2
Sep 17 21:23:03.251797 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:03.251779    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerDied","Data":"da62a611fe2a09bc14813147592400d00529643b1fc84c525e2ed27446226596"}
Sep 17 21:23:04.147378 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:04.147321    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 21:23:08.132878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.132845    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"]
Sep 17 21:23:08.135257 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.135238    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.137761 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.137740    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 21:23:08.137761 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.137753    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:23:08.149147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.149122    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"]
Sep 17 21:23:08.152777 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.152750    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.152873 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.152778    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.152873 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.152798    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p248\" (UniqueName: \"kubernetes.io/projected/06466fca-6abd-44aa-9b52-b5d982019100-kube-api-access-5p248\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.152873 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.152859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06466fca-6abd-44aa-9b52-b5d982019100-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.152991 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.152933    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.253609 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.253580    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.253609 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.253609    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.253772 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.253628    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p248\" (UniqueName: \"kubernetes.io/projected/06466fca-6abd-44aa-9b52-b5d982019100-kube-api-access-5p248\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.253831 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.253768    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06466fca-6abd-44aa-9b52-b5d982019100-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.253891 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.253830    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.253944 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:23:08.253922    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 21:23:08.254010 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:23:08.253998    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls podName:06466fca-6abd-44aa-9b52-b5d982019100 nodeName:}" failed. No retries permitted until 2026-09-17 21:23:08.753976284 +0000 UTC m=+2004.287916095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" (UID: "06466fca-6abd-44aa-9b52-b5d982019100") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 21:23:08.254138 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.254118    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06466fca-6abd-44aa-9b52-b5d982019100-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.254268 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.254251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.254305 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.254289    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.262847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.262828    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p248\" (UniqueName: \"kubernetes.io/projected/06466fca-6abd-44aa-9b52-b5d982019100-kube-api-access-5p248\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.758076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.758043    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:08.760389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:08.760349    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:09.044826 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:09.044743    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:09.146944 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:09.146897    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 21:23:09.315293 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:09.315270    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"]
Sep 17 21:23:09.317901 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:23:09.317875    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06466fca_6abd_44aa_9b52_b5d982019100.slice/crio-a59f250d044d2d5d13bbeb18f88016c5a7a7293dff8419c87a44a3ce34ff2c8d WatchSource:0}: Error finding container a59f250d044d2d5d13bbeb18f88016c5a7a7293dff8419c87a44a3ce34ff2c8d: Status 404 returned error can't find the container with id a59f250d044d2d5d13bbeb18f88016c5a7a7293dff8419c87a44a3ce34ff2c8d
Sep 17 21:23:10.279657 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:10.279621    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerStarted","Data":"5f1a096c94df4d7d40f329f6a1a09911e83fdb247aff9f62071aad5ccb051ccf"}
Sep 17 21:23:10.280117 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:10.279680    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerStarted","Data":"a59f250d044d2d5d13bbeb18f88016c5a7a7293dff8419c87a44a3ce34ff2c8d"}
Sep 17 21:23:14.146674 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:14.146643    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 21:23:14.147042 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:14.146754    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:23:14.295928 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:14.295895    2586 generic.go:356] "Generic (PLEG): container finished" podID="06466fca-6abd-44aa-9b52-b5d982019100" containerID="5f1a096c94df4d7d40f329f6a1a09911e83fdb247aff9f62071aad5ccb051ccf" exitCode=0
Sep 17 21:23:14.296095 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:14.295972    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerDied","Data":"5f1a096c94df4d7d40f329f6a1a09911e83fdb247aff9f62071aad5ccb051ccf"}
Sep 17 21:23:15.301405 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:15.301373    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerStarted","Data":"7e984d964e8975ec6ad4e24ec87dcb09fe17748944f999804e3b1f6a89485144"}
Sep 17 21:23:15.301771 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:15.301439    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerStarted","Data":"66c990cf2a4a6fff885d705844d7783bbc90db50663a8dbbff1616bafca3e259"}
Sep 17 21:23:15.301771 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:15.301638    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:15.322477 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:15.322435    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podStartSLOduration=3.011061788 podStartE2EDuration="7.322419995s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.311358207s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:23:15.320443043 +0000 UTC m=+2010.854382852" watchObservedRunningTime="2026-09-17 21:23:15.322419995 +0000 UTC m=+2010.856359803"
Sep 17 21:23:16.304993 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:16.304960    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:16.306197 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:16.306173    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 21:23:17.308925 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:17.308885    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 21:23:19.147218 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:19.147178    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 21:23:22.314469 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:22.314434    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:22.315376 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:22.315324    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 21:23:24.147426 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:24.147386    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 21:23:29.147098 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:29.147053    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 21:23:32.315910 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:32.315879    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:23:33.369590 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.369557    2586 generic.go:356] "Generic (PLEG): container finished" podID="2368e074-19ab-4cc5-a772-b08c24b35660" containerID="ddc5e0cd6f88d8da634a314d91e4201e5864d5648dfd614d536be4110bc55c7e" exitCode=137
Sep 17 21:23:33.369945 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.369600    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerDied","Data":"ddc5e0cd6f88d8da634a314d91e4201e5864d5648dfd614d536be4110bc55c7e"}
Sep 17 21:23:33.582458 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.582434    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:23:33.642292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.642220    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-kube-rbac-proxy-sar-config\") pod \"2368e074-19ab-4cc5-a772-b08c24b35660\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") "
Sep 17 21:23:33.642441 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.642257    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjlzq\" (UniqueName: \"kubernetes.io/projected/2368e074-19ab-4cc5-a772-b08c24b35660-kube-api-access-wjlzq\") pod \"2368e074-19ab-4cc5-a772-b08c24b35660\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") "
Sep 17 21:23:33.642441 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.642384    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls\") pod \"2368e074-19ab-4cc5-a772-b08c24b35660\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") "
Sep 17 21:23:33.642441 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.642417    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2368e074-19ab-4cc5-a772-b08c24b35660-kserve-provision-location\") pod \"2368e074-19ab-4cc5-a772-b08c24b35660\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") "
Sep 17 21:23:33.642603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.642445    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-cabundle-cert\") pod \"2368e074-19ab-4cc5-a772-b08c24b35660\" (UID: \"2368e074-19ab-4cc5-a772-b08c24b35660\") "
Sep 17 21:23:33.642603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.642578    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2368e074-19ab-4cc5-a772-b08c24b35660" (UID: "2368e074-19ab-4cc5-a772-b08c24b35660"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:23:33.642864 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.642843    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2368e074-19ab-4cc5-a772-b08c24b35660" (UID: "2368e074-19ab-4cc5-a772-b08c24b35660"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:23:33.644383 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.644335    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2368e074-19ab-4cc5-a772-b08c24b35660" (UID: "2368e074-19ab-4cc5-a772-b08c24b35660"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:23:33.644482 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.644392    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2368e074-19ab-4cc5-a772-b08c24b35660-kube-api-access-wjlzq" (OuterVolumeSpecName: "kube-api-access-wjlzq") pod "2368e074-19ab-4cc5-a772-b08c24b35660" (UID: "2368e074-19ab-4cc5-a772-b08c24b35660"). InnerVolumeSpecName "kube-api-access-wjlzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:23:33.653370 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.653334    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2368e074-19ab-4cc5-a772-b08c24b35660-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2368e074-19ab-4cc5-a772-b08c24b35660" (UID: "2368e074-19ab-4cc5-a772-b08c24b35660"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:23:33.743388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.743345    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2368e074-19ab-4cc5-a772-b08c24b35660-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:23:33.743388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.743385    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2368e074-19ab-4cc5-a772-b08c24b35660-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:23:33.743528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.743395    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:23:33.743528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.743404    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2368e074-19ab-4cc5-a772-b08c24b35660-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:23:33.743528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:33.743412    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjlzq\" (UniqueName: \"kubernetes.io/projected/2368e074-19ab-4cc5-a772-b08c24b35660-kube-api-access-wjlzq\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:23:34.375673 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.375639    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2" event={"ID":"2368e074-19ab-4cc5-a772-b08c24b35660","Type":"ContainerDied","Data":"8466c2e3b2885de98832402635f6e2c4cf00ec5b498f26b7454372d53eb429de"}
Sep 17 21:23:34.376032 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.375689    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"
Sep 17 21:23:34.376032 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.375733    2586 scope.go:117] "RemoveContainer" containerID="da62a611fe2a09bc14813147592400d00529643b1fc84c525e2ed27446226596"
Sep 17 21:23:34.384438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.384417    2586 scope.go:117] "RemoveContainer" containerID="ddc5e0cd6f88d8da634a314d91e4201e5864d5648dfd614d536be4110bc55c7e"
Sep 17 21:23:34.391532 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.391516    2586 scope.go:117] "RemoveContainer" containerID="e3af4fea3b0ef9b8aa059e937b3b81abec80b17e0fbe2247781b7aa1612f8497"
Sep 17 21:23:34.398579 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.398503    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"]
Sep 17 21:23:34.402813 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.402793    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-rlzq2"]
Sep 17 21:23:34.670930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:34.670855    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" path="/var/lib/kubelet/pods/2368e074-19ab-4cc5-a772-b08c24b35660/volumes"
Sep 17 21:23:48.048438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:48.048407    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"]
Sep 17 21:23:48.049128 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:48.048670    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kserve-container" containerID="cri-o://66c990cf2a4a6fff885d705844d7783bbc90db50663a8dbbff1616bafca3e259" gracePeriod=30
Sep 17 21:23:48.049128 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:48.048725    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy" containerID="cri-o://7e984d964e8975ec6ad4e24ec87dcb09fe17748944f999804e3b1f6a89485144" gracePeriod=30
Sep 17 21:23:48.432049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:48.432020    2586 generic.go:356] "Generic (PLEG): container finished" podID="06466fca-6abd-44aa-9b52-b5d982019100" containerID="7e984d964e8975ec6ad4e24ec87dcb09fe17748944f999804e3b1f6a89485144" exitCode=2
Sep 17 21:23:48.432202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:48.432055    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerDied","Data":"7e984d964e8975ec6ad4e24ec87dcb09fe17748944f999804e3b1f6a89485144"}
Sep 17 21:23:52.311487 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:52.311452    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 21:23:53.234104 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234068    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"]
Sep 17 21:23:53.234321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234309    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kserve-container"
Sep 17 21:23:53.234390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234323    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kserve-container"
Sep 17 21:23:53.234390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234339    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy"
Sep 17 21:23:53.234390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234345    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy"
Sep 17 21:23:53.234390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234376    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="storage-initializer"
Sep 17 21:23:53.234390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234383    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="storage-initializer"
Sep 17 21:23:53.234545 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234426    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kube-rbac-proxy"
Sep 17 21:23:53.234545 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.234434    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2368e074-19ab-4cc5-a772-b08c24b35660" containerName="kserve-container"
Sep 17 21:23:53.238448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.238431    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.240907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.240885    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 21:23:53.240907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.240900    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 21:23:53.248726 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.248705    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"]
Sep 17 21:23:53.380028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.379996    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851a6d07-787e-43bd-8d87-80b3078a5844-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.380424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.380037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.380424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.380053    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.380424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.380089    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/851a6d07-787e-43bd-8d87-80b3078a5844-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.380424 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.380160    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkkx\" (UniqueName: \"kubernetes.io/projected/851a6d07-787e-43bd-8d87-80b3078a5844-kube-api-access-jrkkx\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.480972 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.480945    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/851a6d07-787e-43bd-8d87-80b3078a5844-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.481117 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.480984    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkkx\" (UniqueName: \"kubernetes.io/projected/851a6d07-787e-43bd-8d87-80b3078a5844-kube-api-access-jrkkx\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.481117 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.481053    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851a6d07-787e-43bd-8d87-80b3078a5844-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.481117 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.481099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.481117 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.481115    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.481591 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.481571    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851a6d07-787e-43bd-8d87-80b3078a5844-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.481786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.481770    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.481826 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.481802    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.483730 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.483713    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/851a6d07-787e-43bd-8d87-80b3078a5844-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.490318 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.490272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkkx\" (UniqueName: \"kubernetes.io/projected/851a6d07-787e-43bd-8d87-80b3078a5844-kube-api-access-jrkkx\") pod \"isvc-triton-predictor-5d7db58676-xp4rh\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.548896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.548865    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:23:53.835028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:53.834945    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"]
Sep 17 21:23:53.838973 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:23:53.838947    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851a6d07_787e_43bd_8d87_80b3078a5844.slice/crio-ccb60e0d88457a7569bb7616d330f62a5c7bc0e02c753fc0bfaf2b0af207483a WatchSource:0}: Error finding container ccb60e0d88457a7569bb7616d330f62a5c7bc0e02c753fc0bfaf2b0af207483a: Status 404 returned error can't find the container with id ccb60e0d88457a7569bb7616d330f62a5c7bc0e02c753fc0bfaf2b0af207483a
Sep 17 21:23:54.454108 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:54.454071    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerStarted","Data":"89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb"}
Sep 17 21:23:54.454505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:54.454128    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerStarted","Data":"ccb60e0d88457a7569bb7616d330f62a5c7bc0e02c753fc0bfaf2b0af207483a"}
Sep 17 21:23:57.309547 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:57.309459    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 21:23:58.470311 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:58.470271    2586 generic.go:356] "Generic (PLEG): container finished" podID="851a6d07-787e-43bd-8d87-80b3078a5844" containerID="89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb" exitCode=0
Sep 17 21:23:58.470686 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:23:58.470329    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerDied","Data":"89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb"}
Sep 17 21:24:02.309533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:02.309477    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 21:24:02.310014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:02.309694    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:24:05.044107 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:05.044063    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"]
Sep 17 21:24:05.044588 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:05.044348    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" containerID="cri-o://43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882" gracePeriod=30
Sep 17 21:24:05.045078 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:05.044741    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kube-rbac-proxy" containerID="cri-o://74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8" gracePeriod=30
Sep 17 21:24:06.773129 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:06.773083    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 21:24:06.777422 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:06.777389    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 21:24:07.310378 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:07.310200    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 21:24:07.547615 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:07.547566    2586 generic.go:356] "Generic (PLEG): container finished" podID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerID="74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8" exitCode=2
Sep 17 21:24:07.547851 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:07.547692    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerDied","Data":"74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8"}
Sep 17 21:24:10.206532 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.204880    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"]
Sep 17 21:24:10.209453 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.208712    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.212581 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.212244    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 21:24:10.213695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.213182    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 21:24:10.219349 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.219274    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"]
Sep 17 21:24:10.292323 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.291931    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"]
Sep 17 21:24:10.296527 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.295655    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.298531 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.298285    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 21:24:10.307594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.307572    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"]
Sep 17 21:24:10.326375 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.326331    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.326575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.326398    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.326575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.326426    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.326575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.326475    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlt7\" (UniqueName: \"kubernetes.io/projected/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-api-access-mtlt7\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.326575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.326500    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.427835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.427799    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cfc78c-f47d-4303-9b2c-f0678648cf01-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.428014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.427846    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.428014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.427884    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlt7\" (UniqueName: \"kubernetes.io/projected/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-api-access-mtlt7\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.428014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.427906    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cfc78c-f47d-4303-9b2c-f0678648cf01-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.428014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.427933    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.428014 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.427988    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.428283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.428016    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.428283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.428057    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.428283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.428085    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-api-access-zqphq\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.428283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.428112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.429177 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.429149    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.430081 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.430057    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.430888 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.430783    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.441037 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.433800    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.441037 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.437029    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlt7\" (UniqueName: \"kubernetes.io/projected/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-api-access-mtlt7\") pod \"isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.528893 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.528810    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.528893 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.528872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cfc78c-f47d-4303-9b2c-f0678648cf01-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.529123 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.528933    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.529123 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.528979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-api-access-zqphq\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.529123 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.529014    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cfc78c-f47d-4303-9b2c-f0678648cf01-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.529417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.529391    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cfc78c-f47d-4303-9b2c-f0678648cf01-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.530101 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.530019    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.530528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.530499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.532446 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.532421    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:10.544997 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.544941    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cfc78c-f47d-4303-9b2c-f0678648cf01-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.549686 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.549643    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-api-access-zqphq\") pod \"isvc-canary-rollback-predictor-97977b6b5-v4rkc\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.610337 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.610302    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:10.848256 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.847717    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"]
Sep 17 21:24:10.851181 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:24:10.851152    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf6f232_3434_43f4_a878_bd5bd3fcfec9.slice/crio-a93790019c3aac787badd69ffd9dec2e08e0092bf79d8adc79cd12e1b9e8f2b4 WatchSource:0}: Error finding container a93790019c3aac787badd69ffd9dec2e08e0092bf79d8adc79cd12e1b9e8f2b4: Status 404 returned error can't find the container with id a93790019c3aac787badd69ffd9dec2e08e0092bf79d8adc79cd12e1b9e8f2b4
Sep 17 21:24:10.915798 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.915695    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:24:10.925656 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:10.925581    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"]
Sep 17 21:24:10.926339 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:24:10.926306    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cfc78c_f47d_4303_9b2c_f0678648cf01.slice/crio-e7423e0e1cb60daf2543cc2c46e45ee8c1532d1a6a2b2bf6bdc0685b8f5fc36e WatchSource:0}: Error finding container e7423e0e1cb60daf2543cc2c46e45ee8c1532d1a6a2b2bf6bdc0685b8f5fc36e: Status 404 returned error can't find the container with id e7423e0e1cb60daf2543cc2c46e45ee8c1532d1a6a2b2bf6bdc0685b8f5fc36e
Sep 17 21:24:11.035838 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.035806    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-rbac-proxy-sar-config\") pod \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") "
Sep 17 21:24:11.035969 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.035856    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-cabundle-cert\") pod \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") "
Sep 17 21:24:11.035969 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.035903    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kserve-provision-location\") pod \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") "
Sep 17 21:24:11.035969 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.035938    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65c3cc9-2338-43df-a900-a257b4d6f2ff-proxy-tls\") pod \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") "
Sep 17 21:24:11.036145 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.036008    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l24x4\" (UniqueName: \"kubernetes.io/projected/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-api-access-l24x4\") pod \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\" (UID: \"e65c3cc9-2338-43df-a900-a257b4d6f2ff\") "
Sep 17 21:24:11.036957 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.036928    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e65c3cc9-2338-43df-a900-a257b4d6f2ff" (UID: "e65c3cc9-2338-43df-a900-a257b4d6f2ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:24:11.037098 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.037076    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e65c3cc9-2338-43df-a900-a257b4d6f2ff" (UID: "e65c3cc9-2338-43df-a900-a257b4d6f2ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:11.037171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.037093    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e65c3cc9-2338-43df-a900-a257b4d6f2ff" (UID: "e65c3cc9-2338-43df-a900-a257b4d6f2ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:24:11.043390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.043341    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65c3cc9-2338-43df-a900-a257b4d6f2ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e65c3cc9-2338-43df-a900-a257b4d6f2ff" (UID: "e65c3cc9-2338-43df-a900-a257b4d6f2ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:24:11.046403 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.046374    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-api-access-l24x4" (OuterVolumeSpecName: "kube-api-access-l24x4") pod "e65c3cc9-2338-43df-a900-a257b4d6f2ff" (UID: "e65c3cc9-2338-43df-a900-a257b4d6f2ff"). InnerVolumeSpecName "kube-api-access-l24x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:24:11.136959 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.136922    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:11.137139 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.136963    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65c3cc9-2338-43df-a900-a257b4d6f2ff-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:11.137139 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.136982    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:11.137139 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.136996    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65c3cc9-2338-43df-a900-a257b4d6f2ff-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:11.137139 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.137010    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24x4\" (UniqueName: \"kubernetes.io/projected/e65c3cc9-2338-43df-a900-a257b4d6f2ff-kube-api-access-l24x4\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:11.578423 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.577708    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerStarted","Data":"b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917"}
Sep 17 21:24:11.578423 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.577782    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerStarted","Data":"e7423e0e1cb60daf2543cc2c46e45ee8c1532d1a6a2b2bf6bdc0685b8f5fc36e"}
Sep 17 21:24:11.581197 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.581102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerStarted","Data":"6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c"}
Sep 17 21:24:11.581197 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.581159    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerStarted","Data":"a93790019c3aac787badd69ffd9dec2e08e0092bf79d8adc79cd12e1b9e8f2b4"}
Sep 17 21:24:11.583772 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.583697    2586 generic.go:356] "Generic (PLEG): container finished" podID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerID="43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882" exitCode=0
Sep 17 21:24:11.583772 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.583734    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerDied","Data":"43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882"}
Sep 17 21:24:11.583772 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.583751    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"
Sep 17 21:24:11.583986 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.583811    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7" event={"ID":"e65c3cc9-2338-43df-a900-a257b4d6f2ff","Type":"ContainerDied","Data":"61258dd8358e7a6fa276d59a94cf82b553954d291a0ce643304bce61170055b1"}
Sep 17 21:24:11.583986 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.583851    2586 scope.go:117] "RemoveContainer" containerID="74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8"
Sep 17 21:24:11.598078 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.597950    2586 scope.go:117] "RemoveContainer" containerID="43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882"
Sep 17 21:24:11.610940 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.610850    2586 scope.go:117] "RemoveContainer" containerID="36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39"
Sep 17 21:24:11.624834 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.624709    2586 scope.go:117] "RemoveContainer" containerID="74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8"
Sep 17 21:24:11.625137 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:24:11.625112    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8\": container with ID starting with 74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8 not found: ID does not exist" containerID="74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8"
Sep 17 21:24:11.625226 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.625148    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8"} err="failed to get container status \"74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8\": rpc error: code = NotFound desc = could not find container \"74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8\": container with ID starting with 74a05316b2c7d76a49fda44c068d2b456ce69a2882e417d9051d9c7ba120a8b8 not found: ID does not exist"
Sep 17 21:24:11.625226 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.625171    2586 scope.go:117] "RemoveContainer" containerID="43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882"
Sep 17 21:24:11.625507 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:24:11.625465    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882\": container with ID starting with 43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882 not found: ID does not exist" containerID="43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882"
Sep 17 21:24:11.625507 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.625493    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882"} err="failed to get container status \"43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882\": rpc error: code = NotFound desc = could not find container \"43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882\": container with ID starting with 43d91ae852c9ec84da6d9c694e4ead93f8e9079fc0cb772d4eb28e0084757882 not found: ID does not exist"
Sep 17 21:24:11.625705 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.625514    2586 scope.go:117] "RemoveContainer" containerID="36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39"
Sep 17 21:24:11.625879 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:24:11.625774    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39\": container with ID starting with 36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39 not found: ID does not exist" containerID="36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39"
Sep 17 21:24:11.625879 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.625807    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39"} err="failed to get container status \"36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39\": rpc error: code = NotFound desc = could not find container \"36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39\": container with ID starting with 36893669d903a6b6dbb07adccd689f38f8a70a02c41098e1acbd9903c6eb4b39 not found: ID does not exist"
Sep 17 21:24:11.637597 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.637571    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"]
Sep 17 21:24:11.640652 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:11.640621    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-x8zn7"]
Sep 17 21:24:12.309834 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:12.309789    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 21:24:12.673209 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:12.673168    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" path="/var/lib/kubelet/pods/e65c3cc9-2338-43df-a900-a257b4d6f2ff/volumes"
Sep 17 21:24:15.607415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:15.607374    2586 generic.go:356] "Generic (PLEG): container finished" podID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerID="b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917" exitCode=0
Sep 17 21:24:15.607930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:15.607440    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerDied","Data":"b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917"}
Sep 17 21:24:15.609780 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:15.609668    2586 generic.go:356] "Generic (PLEG): container finished" podID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerID="6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c" exitCode=0
Sep 17 21:24:15.609780 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:15.609713    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerDied","Data":"6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c"}
Sep 17 21:24:16.621801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.620954    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerStarted","Data":"f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d"}
Sep 17 21:24:16.621801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.621029    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerStarted","Data":"a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c"}
Sep 17 21:24:16.621801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.621537    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:16.621801 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.621623    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:16.623159 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.623120    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:24:16.625249 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.625227    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerStarted","Data":"ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273"}
Sep 17 21:24:16.625386 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.625284    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerStarted","Data":"c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60"}
Sep 17 21:24:16.625537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.625514    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:16.625619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.625590    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:16.626711 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.626679    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:24:16.644889 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.644843    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podStartSLOduration=2.469383026 podStartE2EDuration="6.644824943s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.175441917s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:24:16.642071537 +0000 UTC m=+2072.176011336" watchObservedRunningTime="2026-09-17 21:24:16.644824943 +0000 UTC m=+2072.178764760"
Sep 17 21:24:16.663848 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:16.663800    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podStartSLOduration=2.647787261 podStartE2EDuration="6.663783785s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.015996524s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:24:16.661345388 +0000 UTC m=+2072.195285202" watchObservedRunningTime="2026-09-17 21:24:16.663783785 +0000 UTC m=+2072.197723595"
Sep 17 21:24:17.309201 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:17.309156    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 21:24:17.631764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:17.631722    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:24:17.631764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:17.631763    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:24:18.641841 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.641753    2586 generic.go:356] "Generic (PLEG): container finished" podID="06466fca-6abd-44aa-9b52-b5d982019100" containerID="66c990cf2a4a6fff885d705844d7783bbc90db50663a8dbbff1616bafca3e259" exitCode=137
Sep 17 21:24:18.642296 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.641893    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerDied","Data":"66c990cf2a4a6fff885d705844d7783bbc90db50663a8dbbff1616bafca3e259"}
Sep 17 21:24:18.769705 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.769642    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:24:18.905167 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.905125    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06466fca-6abd-44aa-9b52-b5d982019100-kserve-provision-location\") pod \"06466fca-6abd-44aa-9b52-b5d982019100\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") "
Sep 17 21:24:18.905377 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.905178    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-kube-rbac-proxy-sar-config\") pod \"06466fca-6abd-44aa-9b52-b5d982019100\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") "
Sep 17 21:24:18.905377 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.905217    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls\") pod \"06466fca-6abd-44aa-9b52-b5d982019100\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") "
Sep 17 21:24:18.905377 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.905244    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-cabundle-cert\") pod \"06466fca-6abd-44aa-9b52-b5d982019100\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") "
Sep 17 21:24:18.905377 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.905269    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p248\" (UniqueName: \"kubernetes.io/projected/06466fca-6abd-44aa-9b52-b5d982019100-kube-api-access-5p248\") pod \"06466fca-6abd-44aa-9b52-b5d982019100\" (UID: \"06466fca-6abd-44aa-9b52-b5d982019100\") "
Sep 17 21:24:18.905906 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.905860    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "06466fca-6abd-44aa-9b52-b5d982019100" (UID: "06466fca-6abd-44aa-9b52-b5d982019100"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:24:18.906027 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.906003    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "06466fca-6abd-44aa-9b52-b5d982019100" (UID: "06466fca-6abd-44aa-9b52-b5d982019100"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:24:18.908028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.907989    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06466fca-6abd-44aa-9b52-b5d982019100-kube-api-access-5p248" (OuterVolumeSpecName: "kube-api-access-5p248") pod "06466fca-6abd-44aa-9b52-b5d982019100" (UID: "06466fca-6abd-44aa-9b52-b5d982019100"). InnerVolumeSpecName "kube-api-access-5p248". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:24:18.908891 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.908854    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "06466fca-6abd-44aa-9b52-b5d982019100" (UID: "06466fca-6abd-44aa-9b52-b5d982019100"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:24:18.918318 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:18.915629    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06466fca-6abd-44aa-9b52-b5d982019100-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06466fca-6abd-44aa-9b52-b5d982019100" (UID: "06466fca-6abd-44aa-9b52-b5d982019100"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:24:19.005878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.005850    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06466fca-6abd-44aa-9b52-b5d982019100-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:19.005878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.005880    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:19.006101 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.005895    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/06466fca-6abd-44aa-9b52-b5d982019100-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:19.006101 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.005908    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/06466fca-6abd-44aa-9b52-b5d982019100-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:19.006101 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.005919    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p248\" (UniqueName: \"kubernetes.io/projected/06466fca-6abd-44aa-9b52-b5d982019100-kube-api-access-5p248\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:24:19.650079 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.650039    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz" event={"ID":"06466fca-6abd-44aa-9b52-b5d982019100","Type":"ContainerDied","Data":"a59f250d044d2d5d13bbeb18f88016c5a7a7293dff8419c87a44a3ce34ff2c8d"}
Sep 17 21:24:19.650569 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.650127    2586 scope.go:117] "RemoveContainer" containerID="7e984d964e8975ec6ad4e24ec87dcb09fe17748944f999804e3b1f6a89485144"
Sep 17 21:24:19.650569 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.650288    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"
Sep 17 21:24:19.661962 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.661900    2586 scope.go:117] "RemoveContainer" containerID="66c990cf2a4a6fff885d705844d7783bbc90db50663a8dbbff1616bafca3e259"
Sep 17 21:24:19.671277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.671256    2586 scope.go:117] "RemoveContainer" containerID="5f1a096c94df4d7d40f329f6a1a09911e83fdb247aff9f62071aad5ccb051ccf"
Sep 17 21:24:19.678046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.678022    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"]
Sep 17 21:24:19.680920 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:19.680900    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-rbnjz"]
Sep 17 21:24:20.671965 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:20.671849    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06466fca-6abd-44aa-9b52-b5d982019100" path="/var/lib/kubelet/pods/06466fca-6abd-44aa-9b52-b5d982019100/volumes"
Sep 17 21:24:22.637383 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:22.637276    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:24:22.638008 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:22.637865    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:24:22.638147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:22.638122    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:24:22.638733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:22.638675    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:24:32.638329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:32.638279    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:24:32.638971 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:32.638932    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:24:42.638885 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:42.638842    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:24:42.639484 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:42.638842    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:24:52.638868 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:52.638823    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:24:52.639319 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:24:52.638822    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:25:02.638270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:02.638231    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:25:02.638831 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:02.638551    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:25:12.637937 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:12.637890    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:25:12.639142 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:12.639100    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:25:22.638319 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:22.638277    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:25:22.638942 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:22.638575    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:25:32.638016 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:32.637929    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:25:32.639191 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:32.639158    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:25:42.638175 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:42.638118    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:25:42.638692 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:42.638591    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:25:48.666626 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:48.666588    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 21:25:51.667219 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:51.667174    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 21:25:52.463321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:52.463295    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:25:52.463515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:52.463501    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:25:54.072691 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:54.072655    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerStarted","Data":"e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd"}
Sep 17 21:25:54.073134 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:54.072711    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerStarted","Data":"7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60"}
Sep 17 21:25:54.073134 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:54.072754    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:25:54.102749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:54.102705    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" podStartSLOduration=2.366716372 podStartE2EDuration="2m1.102692226s" totalImagesPullingTime="1m54.843573776s" totalInitContainerRuntime="3.892402078s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:25:54.100384374 +0000 UTC m=+2169.634324184" watchObservedRunningTime="2026-09-17 21:25:54.102692226 +0000 UTC m=+2169.636632035"
Sep 17 21:25:55.076742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:55.076708    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:25:55.077857 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:55.077831    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 21:25:56.080704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:56.080668    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 21:25:58.670443 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:58.670408    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:25:58.800696 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:58.800667    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"]
Sep 17 21:25:58.800894 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:58.800870    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container" containerID="cri-o://a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c" gracePeriod=30
Sep 17 21:25:58.800983 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:58.800937    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kube-rbac-proxy" containerID="cri-o://f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d" gracePeriod=30
Sep 17 21:25:58.801720 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:58.801508    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:25:59.092727 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:59.092697    2586 generic.go:356] "Generic (PLEG): container finished" podID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerID="f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d" exitCode=2
Sep 17 21:25:59.092877 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:25:59.092741    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerDied","Data":"f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d"}
Sep 17 21:26:00.304792 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.304757    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"]
Sep 17 21:26:00.305339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.304988    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container" containerID="cri-o://c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60" gracePeriod=30
Sep 17 21:26:00.305339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.305017    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kube-rbac-proxy" containerID="cri-o://ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273" gracePeriod=30
Sep 17 21:26:00.385860 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.385829    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"]
Sep 17 21:26:00.386113 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386100    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container"
Sep 17 21:26:00.386157 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386115    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container"
Sep 17 21:26:00.386157 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386127    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kserve-container"
Sep 17 21:26:00.386157 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386134    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kserve-container"
Sep 17 21:26:00.386157 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386143    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kube-rbac-proxy"
Sep 17 21:26:00.386157 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386149    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kube-rbac-proxy"
Sep 17 21:26:00.386157 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386156    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="storage-initializer"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386162    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="storage-initializer"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386170    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="storage-initializer"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386176    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="storage-initializer"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386184    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386189    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386236    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kube-rbac-proxy"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386246    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kserve-container"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386251    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65c3cc9-2338-43df-a900-a257b4d6f2ff" containerName="kube-rbac-proxy"
Sep 17 21:26:00.386340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.386258    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="06466fca-6abd-44aa-9b52-b5d982019100" containerName="kserve-container"
Sep 17 21:26:00.400457 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.400433    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"]
Sep 17 21:26:00.400586 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.400545    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.551072 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.551040    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.551072 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.551073    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.551385 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.551162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.551385 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.551216    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.551385 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.551245    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wtp\" (UniqueName: \"kubernetes.io/projected/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-api-access-t4wtp\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.651945 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.651915    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.652158 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.651954    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.652158 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.651973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wtp\" (UniqueName: \"kubernetes.io/projected/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-api-access-t4wtp\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.652158 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.652041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.652158 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.652076    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.652343 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.652326    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.652751 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.652733    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.652810 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.652758    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.654477 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.654460    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.661116 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.661097    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wtp\" (UniqueName: \"kubernetes.io/projected/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-api-access-t4wtp\") pod \"isvc-canary-rollback-predictor-97977b6b5-4trbg\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.710958 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.710928    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:00.995375 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:00.995332    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"]
Sep 17 21:26:00.997122 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:26:00.997095    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode488f1bd_6e6c_4597_93d6_f65ede8da1c1.slice/crio-19e78226583897894b38da3de776d4ee8cb89f91cca77504cb016ed8be597d62 WatchSource:0}: Error finding container 19e78226583897894b38da3de776d4ee8cb89f91cca77504cb016ed8be597d62: Status 404 returned error can't find the container with id 19e78226583897894b38da3de776d4ee8cb89f91cca77504cb016ed8be597d62
Sep 17 21:26:01.085402 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:01.085380    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:26:01.086064 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:01.086043    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:26:01.102549 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:01.102529    2586 generic.go:356] "Generic (PLEG): container finished" podID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerID="ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273" exitCode=2
Sep 17 21:26:01.102619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:01.102604    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerDied","Data":"ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273"}
Sep 17 21:26:01.104278 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:01.104253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerStarted","Data":"19e78226583897894b38da3de776d4ee8cb89f91cca77504cb016ed8be597d62"}
Sep 17 21:26:02.109657 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:02.109620    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerStarted","Data":"58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc"}
Sep 17 21:26:02.632194 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:02.632154    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 21:26:02.632389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:02.632156    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 21:26:03.475246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.475181    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:26:03.573264 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573232    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-rbac-proxy-sar-config\") pod \"37cfc78c-f47d-4303-9b2c-f0678648cf01\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") "
Sep 17 21:26:03.573466 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573274    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cfc78c-f47d-4303-9b2c-f0678648cf01-kserve-provision-location\") pod \"37cfc78c-f47d-4303-9b2c-f0678648cf01\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") "
Sep 17 21:26:03.573466 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573307    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cfc78c-f47d-4303-9b2c-f0678648cf01-proxy-tls\") pod \"37cfc78c-f47d-4303-9b2c-f0678648cf01\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") "
Sep 17 21:26:03.573466 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573395    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-api-access-zqphq\") pod \"37cfc78c-f47d-4303-9b2c-f0678648cf01\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") "
Sep 17 21:26:03.573466 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573453    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-cabundle-cert\") pod \"37cfc78c-f47d-4303-9b2c-f0678648cf01\" (UID: \"37cfc78c-f47d-4303-9b2c-f0678648cf01\") "
Sep 17 21:26:03.573739 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573650    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cfc78c-f47d-4303-9b2c-f0678648cf01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37cfc78c-f47d-4303-9b2c-f0678648cf01" (UID: "37cfc78c-f47d-4303-9b2c-f0678648cf01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:26:03.573869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573747    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37cfc78c-f47d-4303-9b2c-f0678648cf01" (UID: "37cfc78c-f47d-4303-9b2c-f0678648cf01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:26:03.573869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.573830    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37cfc78c-f47d-4303-9b2c-f0678648cf01" (UID: "37cfc78c-f47d-4303-9b2c-f0678648cf01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:26:03.575425 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.575404    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cfc78c-f47d-4303-9b2c-f0678648cf01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37cfc78c-f47d-4303-9b2c-f0678648cf01" (UID: "37cfc78c-f47d-4303-9b2c-f0678648cf01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:26:03.575504 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.575459    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-api-access-zqphq" (OuterVolumeSpecName: "kube-api-access-zqphq") pod "37cfc78c-f47d-4303-9b2c-f0678648cf01" (UID: "37cfc78c-f47d-4303-9b2c-f0678648cf01"). InnerVolumeSpecName "kube-api-access-zqphq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:26:03.674453 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.674424    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:03.674453 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.674447    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:03.674453 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.674456    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37cfc78c-f47d-4303-9b2c-f0678648cf01-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:03.674663 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.674467    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37cfc78c-f47d-4303-9b2c-f0678648cf01-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:03.674663 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:03.674475    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqphq\" (UniqueName: \"kubernetes.io/projected/37cfc78c-f47d-4303-9b2c-f0678648cf01-kube-api-access-zqphq\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:04.120584 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.120552    2586 generic.go:356] "Generic (PLEG): container finished" podID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerID="a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c" exitCode=0
Sep 17 21:26:04.120748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.120603    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerDied","Data":"a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c"}
Sep 17 21:26:04.120748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.120607    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"
Sep 17 21:26:04.120748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.120651    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc" event={"ID":"37cfc78c-f47d-4303-9b2c-f0678648cf01","Type":"ContainerDied","Data":"e7423e0e1cb60daf2543cc2c46e45ee8c1532d1a6a2b2bf6bdc0685b8f5fc36e"}
Sep 17 21:26:04.120748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.120678    2586 scope.go:117] "RemoveContainer" containerID="f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d"
Sep 17 21:26:04.128745 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.128729    2586 scope.go:117] "RemoveContainer" containerID="a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c"
Sep 17 21:26:04.135685 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.135669    2586 scope.go:117] "RemoveContainer" containerID="b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917"
Sep 17 21:26:04.142341 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.142327    2586 scope.go:117] "RemoveContainer" containerID="f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d"
Sep 17 21:26:04.142648 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:04.142628    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d\": container with ID starting with f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d not found: ID does not exist" containerID="f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d"
Sep 17 21:26:04.142719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.142656    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d"} err="failed to get container status \"f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d\": rpc error: code = NotFound desc = could not find container \"f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d\": container with ID starting with f56a1fd9c27dcbab5e1d46d5079f7bcd5cb7284dbd1a4b2c1e6fbdd1b59a4a1d not found: ID does not exist"
Sep 17 21:26:04.142719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.142673    2586 scope.go:117] "RemoveContainer" containerID="a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c"
Sep 17 21:26:04.142944 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:04.142923    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c\": container with ID starting with a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c not found: ID does not exist" containerID="a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c"
Sep 17 21:26:04.142995 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.142950    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c"} err="failed to get container status \"a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c\": rpc error: code = NotFound desc = could not find container \"a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c\": container with ID starting with a1f610e9eb19bbd7f14ef506cdac9535adfa8bad6c42db3f7ec548359067630c not found: ID does not exist"
Sep 17 21:26:04.142995 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.142972    2586 scope.go:117] "RemoveContainer" containerID="b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917"
Sep 17 21:26:04.143230 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:04.143211    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917\": container with ID starting with b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917 not found: ID does not exist" containerID="b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917"
Sep 17 21:26:04.143287 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.143239    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917"} err="failed to get container status \"b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917\": rpc error: code = NotFound desc = could not find container \"b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917\": container with ID starting with b786f73b9f7335b8fba54259ccb7e321dbd3f3afddbb114d890c7716d63a8917 not found: ID does not exist"
Sep 17 21:26:04.145928 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.145907    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"]
Sep 17 21:26:04.159772 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.159745    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-v4rkc"]
Sep 17 21:26:04.670620 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:04.670590    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" path="/var/lib/kubelet/pods/37cfc78c-f47d-4303-9b2c-f0678648cf01/volumes"
Sep 17 21:26:05.073695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.073671    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:26:05.127374 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.127327    2586 generic.go:356] "Generic (PLEG): container finished" podID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerID="c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60" exitCode=0
Sep 17 21:26:05.127498 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.127397    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"
Sep 17 21:26:05.127545 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.127392    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerDied","Data":"c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60"}
Sep 17 21:26:05.127589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.127559    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz" event={"ID":"8bf6f232-3434-43f4-a878-bd5bd3fcfec9","Type":"ContainerDied","Data":"a93790019c3aac787badd69ffd9dec2e08e0092bf79d8adc79cd12e1b9e8f2b4"}
Sep 17 21:26:05.127651 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.127601    2586 scope.go:117] "RemoveContainer" containerID="ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273"
Sep 17 21:26:05.128937 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.128916    2586 generic.go:356] "Generic (PLEG): container finished" podID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerID="58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc" exitCode=0
Sep 17 21:26:05.129041 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.128941    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerDied","Data":"58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc"}
Sep 17 21:26:05.135346 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.135329    2586 scope.go:117] "RemoveContainer" containerID="c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60"
Sep 17 21:26:05.142601 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.142588    2586 scope.go:117] "RemoveContainer" containerID="6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c"
Sep 17 21:26:05.149241 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.149225    2586 scope.go:117] "RemoveContainer" containerID="ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273"
Sep 17 21:26:05.149523 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:05.149505    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273\": container with ID starting with ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273 not found: ID does not exist" containerID="ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273"
Sep 17 21:26:05.149571 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.149531    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273"} err="failed to get container status \"ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273\": rpc error: code = NotFound desc = could not find container \"ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273\": container with ID starting with ebc31b1d8665c5ee47ae4e3fe3962df65b9a73ac4c208e2eeceaab83ff3ae273 not found: ID does not exist"
Sep 17 21:26:05.149571 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.149547    2586 scope.go:117] "RemoveContainer" containerID="c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60"
Sep 17 21:26:05.149798 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:05.149781    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60\": container with ID starting with c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60 not found: ID does not exist" containerID="c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60"
Sep 17 21:26:05.149859 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.149807    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60"} err="failed to get container status \"c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60\": rpc error: code = NotFound desc = could not find container \"c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60\": container with ID starting with c77a8fcb16e1565a0fcce953d4068383e9625ce14f83171e1291e9219f956d60 not found: ID does not exist"
Sep 17 21:26:05.149859 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.149827    2586 scope.go:117] "RemoveContainer" containerID="6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c"
Sep 17 21:26:05.150070 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:05.150054    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c\": container with ID starting with 6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c not found: ID does not exist" containerID="6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c"
Sep 17 21:26:05.150109 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.150074    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c"} err="failed to get container status \"6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c\": rpc error: code = NotFound desc = could not find container \"6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c\": container with ID starting with 6cd44de879d6ffdcb163434275f1fdc089ddf7db978102b38edb0c5bb9f74d4c not found: ID does not exist"
Sep 17 21:26:05.185266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185211    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-proxy-tls\") pod \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") "
Sep 17 21:26:05.185266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185244    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-rbac-proxy-sar-config\") pod \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") "
Sep 17 21:26:05.185406 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185287    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-cabundle-cert\") pod \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") "
Sep 17 21:26:05.185482 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185401    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kserve-provision-location\") pod \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") "
Sep 17 21:26:05.185482 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185445    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtlt7\" (UniqueName: \"kubernetes.io/projected/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-api-access-mtlt7\") pod \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\" (UID: \"8bf6f232-3434-43f4-a878-bd5bd3fcfec9\") "
Sep 17 21:26:05.185673 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185650    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8bf6f232-3434-43f4-a878-bd5bd3fcfec9" (UID: "8bf6f232-3434-43f4-a878-bd5bd3fcfec9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:26:05.185724 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185656    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8bf6f232-3434-43f4-a878-bd5bd3fcfec9" (UID: "8bf6f232-3434-43f4-a878-bd5bd3fcfec9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:26:05.185724 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.185677    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8bf6f232-3434-43f4-a878-bd5bd3fcfec9" (UID: "8bf6f232-3434-43f4-a878-bd5bd3fcfec9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:26:05.187080 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.187061    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8bf6f232-3434-43f4-a878-bd5bd3fcfec9" (UID: "8bf6f232-3434-43f4-a878-bd5bd3fcfec9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:26:05.187270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.187251    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-api-access-mtlt7" (OuterVolumeSpecName: "kube-api-access-mtlt7") pod "8bf6f232-3434-43f4-a878-bd5bd3fcfec9" (UID: "8bf6f232-3434-43f4-a878-bd5bd3fcfec9"). InnerVolumeSpecName "kube-api-access-mtlt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:26:05.286658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.286634    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:05.286775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.286662    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:05.286775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.286678    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:05.286775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.286691    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:05.286775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.286705    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlt7\" (UniqueName: \"kubernetes.io/projected/8bf6f232-3434-43f4-a878-bd5bd3fcfec9-kube-api-access-mtlt7\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:05.458882 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.458849    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"]
Sep 17 21:26:05.472518 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.472486    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-5db5496f8d-xnwnz"]
Sep 17 21:26:05.773860 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.773790    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"]
Sep 17 21:26:05.774211 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.774028    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kserve-container" containerID="cri-o://7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60" gracePeriod=30
Sep 17 21:26:05.774211 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:05.774052    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kube-rbac-proxy" containerID="cri-o://e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd" gracePeriod=30
Sep 17 21:26:06.081734 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.081660    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 21:26:06.136636 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.136601    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerStarted","Data":"24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd"}
Sep 17 21:26:06.136788 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.136667    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerStarted","Data":"d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d"}
Sep 17 21:26:06.137001 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.136979    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:06.137159 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.137139    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:06.138646 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.138613    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:26:06.138982 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.138961    2586 generic.go:356] "Generic (PLEG): container finished" podID="851a6d07-787e-43bd-8d87-80b3078a5844" containerID="e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd" exitCode=2
Sep 17 21:26:06.139061 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.139002    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerDied","Data":"e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd"}
Sep 17 21:26:06.160872 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.160832    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podStartSLOduration=2.338653129 podStartE2EDuration="6.160818999s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82216587s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:26:06.158488353 +0000 UTC m=+2181.692428203" watchObservedRunningTime="2026-09-17 21:26:06.160818999 +0000 UTC m=+2181.694758807"
Sep 17 21:26:06.671101 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:06.671070    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" path="/var/lib/kubelet/pods/8bf6f232-3434-43f4-a878-bd5bd3fcfec9/volumes"
Sep 17 21:26:07.142488 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:07.142458    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:26:08.334076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.334053    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:26:08.512086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512015    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-kube-rbac-proxy-sar-config\") pod \"851a6d07-787e-43bd-8d87-80b3078a5844\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") "
Sep 17 21:26:08.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512110    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/851a6d07-787e-43bd-8d87-80b3078a5844-proxy-tls\") pod \"851a6d07-787e-43bd-8d87-80b3078a5844\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") "
Sep 17 21:26:08.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512168    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrkkx\" (UniqueName: \"kubernetes.io/projected/851a6d07-787e-43bd-8d87-80b3078a5844-kube-api-access-jrkkx\") pod \"851a6d07-787e-43bd-8d87-80b3078a5844\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") "
Sep 17 21:26:08.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512197    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851a6d07-787e-43bd-8d87-80b3078a5844-kserve-provision-location\") pod \"851a6d07-787e-43bd-8d87-80b3078a5844\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") "
Sep 17 21:26:08.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512221    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-cabundle-cert\") pod \"851a6d07-787e-43bd-8d87-80b3078a5844\" (UID: \"851a6d07-787e-43bd-8d87-80b3078a5844\") "
Sep 17 21:26:08.512553 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512336    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "851a6d07-787e-43bd-8d87-80b3078a5844" (UID: "851a6d07-787e-43bd-8d87-80b3078a5844"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:26:08.512553 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512446    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:08.512662 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512610    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "851a6d07-787e-43bd-8d87-80b3078a5844" (UID: "851a6d07-787e-43bd-8d87-80b3078a5844"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:26:08.512741 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.512715    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851a6d07-787e-43bd-8d87-80b3078a5844-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "851a6d07-787e-43bd-8d87-80b3078a5844" (UID: "851a6d07-787e-43bd-8d87-80b3078a5844"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:26:08.514198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.514175    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/851a6d07-787e-43bd-8d87-80b3078a5844-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "851a6d07-787e-43bd-8d87-80b3078a5844" (UID: "851a6d07-787e-43bd-8d87-80b3078a5844"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:26:08.514260 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.514245    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851a6d07-787e-43bd-8d87-80b3078a5844-kube-api-access-jrkkx" (OuterVolumeSpecName: "kube-api-access-jrkkx") pod "851a6d07-787e-43bd-8d87-80b3078a5844" (UID: "851a6d07-787e-43bd-8d87-80b3078a5844"). InnerVolumeSpecName "kube-api-access-jrkkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:26:08.612824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.612801    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrkkx\" (UniqueName: \"kubernetes.io/projected/851a6d07-787e-43bd-8d87-80b3078a5844-kube-api-access-jrkkx\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:08.612824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.612821    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/851a6d07-787e-43bd-8d87-80b3078a5844-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:08.613001 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.612831    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/851a6d07-787e-43bd-8d87-80b3078a5844-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:08.613001 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:08.612841    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/851a6d07-787e-43bd-8d87-80b3078a5844-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:26:09.152008 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.151975    2586 generic.go:356] "Generic (PLEG): container finished" podID="851a6d07-787e-43bd-8d87-80b3078a5844" containerID="7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60" exitCode=0
Sep 17 21:26:09.152269 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.152032    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerDied","Data":"7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60"}
Sep 17 21:26:09.152269 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.152074    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh" event={"ID":"851a6d07-787e-43bd-8d87-80b3078a5844","Type":"ContainerDied","Data":"ccb60e0d88457a7569bb7616d330f62a5c7bc0e02c753fc0bfaf2b0af207483a"}
Sep 17 21:26:09.152269 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.152097    2586 scope.go:117] "RemoveContainer" containerID="e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd"
Sep 17 21:26:09.152269 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.152033    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"
Sep 17 21:26:09.159735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.159719    2586 scope.go:117] "RemoveContainer" containerID="7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60"
Sep 17 21:26:09.166589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.166573    2586 scope.go:117] "RemoveContainer" containerID="89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb"
Sep 17 21:26:09.170651 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.170629    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"]
Sep 17 21:26:09.173339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.173323    2586 scope.go:117] "RemoveContainer" containerID="e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd"
Sep 17 21:26:09.173601 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:09.173584    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd\": container with ID starting with e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd not found: ID does not exist" containerID="e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd"
Sep 17 21:26:09.173655 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.173610    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd"} err="failed to get container status \"e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd\": rpc error: code = NotFound desc = could not find container \"e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd\": container with ID starting with e721977eb40c3b9974897a2a1566494f090c113c7623aacf9f62693bae9da4cd not found: ID does not exist"
Sep 17 21:26:09.173655 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.173627    2586 scope.go:117] "RemoveContainer" containerID="7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60"
Sep 17 21:26:09.173874 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:09.173859    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60\": container with ID starting with 7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60 not found: ID does not exist" containerID="7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60"
Sep 17 21:26:09.173917 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.173876    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60"} err="failed to get container status \"7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60\": rpc error: code = NotFound desc = could not find container \"7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60\": container with ID starting with 7af854f079b861eb45be774a37310a03d78c46fd57898fab80cdc9c6dffcad60 not found: ID does not exist"
Sep 17 21:26:09.173917 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.173888    2586 scope.go:117] "RemoveContainer" containerID="89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb"
Sep 17 21:26:09.174075 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:26:09.174061    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb\": container with ID starting with 89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb not found: ID does not exist" containerID="89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb"
Sep 17 21:26:09.174154 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.174076    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb"} err="failed to get container status \"89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb\": rpc error: code = NotFound desc = could not find container \"89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb\": container with ID starting with 89c3d4f0e3656ee70081489101e1ff27aa77ff4c9d0170c63cb082a579b537eb not found: ID does not exist"
Sep 17 21:26:09.177983 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:09.177964    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-xp4rh"]
Sep 17 21:26:10.670274 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.670241    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" path="/var/lib/kubelet/pods/851a6d07-787e-43bd-8d87-80b3078a5844/volumes"
Sep 17 21:26:10.952943 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.952872    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"]
Sep 17 21:26:10.953138 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953126    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="storage-initializer"
Sep 17 21:26:10.953192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953140    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="storage-initializer"
Sep 17 21:26:10.953192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953149    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="storage-initializer"
Sep 17 21:26:10.953192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953154    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="storage-initializer"
Sep 17 21:26:10.953192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953162    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kserve-container"
Sep 17 21:26:10.953192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953169    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kserve-container"
Sep 17 21:26:10.953192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953184    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container"
Sep 17 21:26:10.953192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953192    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953199    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="storage-initializer"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953204    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="storage-initializer"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953214    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953221    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953226    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953233    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953240    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953245    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953251    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953258    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953300    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kserve-container"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953308    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kserve-container"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953314    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf6f232-3434-43f4-a878-bd5bd3fcfec9" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953319    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953325    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cfc78c-f47d-4303-9b2c-f0678648cf01" containerName="kube-rbac-proxy"
Sep 17 21:26:10.953430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.953330    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="851a6d07-787e-43bd-8d87-80b3078a5844" containerName="kserve-container"
Sep 17 21:26:10.972607 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.972579    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"]
Sep 17 21:26:10.972748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.972704    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:10.975330 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.975311    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 21:26:10.975768 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:10.975753    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 21:26:11.128913 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.128882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.129082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.128921    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336baf50-870c-431d-8560-463815c21bba-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.129082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.129017    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336baf50-870c-431d-8560-463815c21bba-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.129082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.129050    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.129082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.129076    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm899\" (UniqueName: \"kubernetes.io/projected/336baf50-870c-431d-8560-463815c21bba-kube-api-access-mm899\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.230408 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.230322    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336baf50-870c-431d-8560-463815c21bba-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.230408 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.230383    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336baf50-870c-431d-8560-463815c21bba-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.230408 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.230406    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.230666 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.230429    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm899\" (UniqueName: \"kubernetes.io/projected/336baf50-870c-431d-8560-463815c21bba-kube-api-access-mm899\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.230666 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.230462    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.230806 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.230785    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336baf50-870c-431d-8560-463815c21bba-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.231083 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.231065    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.231142 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.231126    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.232770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.232751    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336baf50-870c-431d-8560-463815c21bba-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.239196 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.239174    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm899\" (UniqueName: \"kubernetes.io/projected/336baf50-870c-431d-8560-463815c21bba-kube-api-access-mm899\") pod \"isvc-xgboost-predictor-7566d65bf7-ldld9\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.283086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.283066    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:11.581179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:11.581154    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"]
Sep 17 21:26:11.583580 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:26:11.583551    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336baf50_870c_431d_8560_463815c21bba.slice/crio-61f495ad7fdf17d2473cdbf61fe0b475f4f55e608783834b61cd2d0b7700fb03 WatchSource:0}: Error finding container 61f495ad7fdf17d2473cdbf61fe0b475f4f55e608783834b61cd2d0b7700fb03: Status 404 returned error can't find the container with id 61f495ad7fdf17d2473cdbf61fe0b475f4f55e608783834b61cd2d0b7700fb03
Sep 17 21:26:12.148934 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:12.148904    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:26:12.149349 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:12.149302    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:26:12.165306 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:12.165276    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerStarted","Data":"5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27"}
Sep 17 21:26:12.165430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:12.165331    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerStarted","Data":"61f495ad7fdf17d2473cdbf61fe0b475f4f55e608783834b61cd2d0b7700fb03"}
Sep 17 21:26:16.182109 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:16.182077    2586 generic.go:356] "Generic (PLEG): container finished" podID="336baf50-870c-431d-8560-463815c21bba" containerID="5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27" exitCode=0
Sep 17 21:26:16.182526 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:16.182154    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerDied","Data":"5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27"}
Sep 17 21:26:22.150143 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:22.150105    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:26:32.150331 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:32.150290    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:26:38.276555 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:38.276518    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerStarted","Data":"0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829"}
Sep 17 21:26:38.276919 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:38.276608    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerStarted","Data":"9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a"}
Sep 17 21:26:38.276919 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:38.276884    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:38.277037 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:38.277021    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:38.278417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:38.278392    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:26:38.297047 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:38.297002    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podStartSLOduration=3.303834775 podStartE2EDuration="28.296975428s" totalImagesPullingTime="21.18364954s" totalInitContainerRuntime="3.809491113s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:26:38.296717944 +0000 UTC m=+2213.830657756" watchObservedRunningTime="2026-09-17 21:26:38.296975428 +0000 UTC m=+2213.830915232"
Sep 17 21:26:39.280750 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:39.280712    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:26:42.149536 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:42.149496    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:26:44.285795 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:44.285768    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:26:44.286242 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:44.286213    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:26:52.149824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:52.149785    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:26:54.286751 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:26:54.286710    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:27:02.150365 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:02.150277    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:27:04.286875 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:04.286838    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:27:12.149622 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:12.149586    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:27:14.286729 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:14.286690    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:27:15.666575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:15.666536    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 21:27:24.286351 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:24.286314    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:27:25.667023 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:25.666995    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:27:30.439884 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:30.439837    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"]
Sep 17 21:27:30.440343 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:30.440170    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container" containerID="cri-o://d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d" gracePeriod=30
Sep 17 21:27:30.440523 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:30.440490    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kube-rbac-proxy" containerID="cri-o://24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd" gracePeriod=30
Sep 17 21:27:31.479463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:31.479432    2586 generic.go:356] "Generic (PLEG): container finished" podID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerID="24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd" exitCode=2
Sep 17 21:27:31.479820 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:31.479471    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerDied","Data":"24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd"}
Sep 17 21:27:32.143458 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:32.143417    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 21:27:34.286843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:34.286801    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:27:34.881194 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:34.881171    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:27:35.003909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.003823    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-rbac-proxy-sar-config\") pod \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") "
Sep 17 21:27:35.003909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.003866    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kserve-provision-location\") pod \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") "
Sep 17 21:27:35.003909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.003892    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-proxy-tls\") pod \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") "
Sep 17 21:27:35.004185 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.004022    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4wtp\" (UniqueName: \"kubernetes.io/projected/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-api-access-t4wtp\") pod \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") "
Sep 17 21:27:35.004185 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.004091    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-cabundle-cert\") pod \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\" (UID: \"e488f1bd-6e6c-4597-93d6-f65ede8da1c1\") "
Sep 17 21:27:35.004287 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.004207    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e488f1bd-6e6c-4597-93d6-f65ede8da1c1" (UID: "e488f1bd-6e6c-4597-93d6-f65ede8da1c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:27:35.004287 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.004207    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e488f1bd-6e6c-4597-93d6-f65ede8da1c1" (UID: "e488f1bd-6e6c-4597-93d6-f65ede8da1c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:35.004430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.004328    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.004430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.004375    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.004506 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.004456    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e488f1bd-6e6c-4597-93d6-f65ede8da1c1" (UID: "e488f1bd-6e6c-4597-93d6-f65ede8da1c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:27:35.006037 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.006017    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e488f1bd-6e6c-4597-93d6-f65ede8da1c1" (UID: "e488f1bd-6e6c-4597-93d6-f65ede8da1c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:27:35.006037 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.006024    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-api-access-t4wtp" (OuterVolumeSpecName: "kube-api-access-t4wtp") pod "e488f1bd-6e6c-4597-93d6-f65ede8da1c1" (UID: "e488f1bd-6e6c-4597-93d6-f65ede8da1c1"). InnerVolumeSpecName "kube-api-access-t4wtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:27:35.105097 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.105070    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.105097 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.105091    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.105097 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.105100    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wtp\" (UniqueName: \"kubernetes.io/projected/e488f1bd-6e6c-4597-93d6-f65ede8da1c1-kube-api-access-t4wtp\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:35.493811 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.493777    2586 generic.go:356] "Generic (PLEG): container finished" podID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerID="d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d" exitCode=0
Sep 17 21:27:35.494202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.493836    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"
Sep 17 21:27:35.494202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.493865    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerDied","Data":"d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d"}
Sep 17 21:27:35.494202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.493925    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg" event={"ID":"e488f1bd-6e6c-4597-93d6-f65ede8da1c1","Type":"ContainerDied","Data":"19e78226583897894b38da3de776d4ee8cb89f91cca77504cb016ed8be597d62"}
Sep 17 21:27:35.494202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.493953    2586 scope.go:117] "RemoveContainer" containerID="24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd"
Sep 17 21:27:35.505375 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.505338    2586 scope.go:117] "RemoveContainer" containerID="d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d"
Sep 17 21:27:35.512648 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.512629    2586 scope.go:117] "RemoveContainer" containerID="58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc"
Sep 17 21:27:35.520773 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.520651    2586 scope.go:117] "RemoveContainer" containerID="24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd"
Sep 17 21:27:35.521290 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:27:35.521016    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd\": container with ID starting with 24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd not found: ID does not exist" containerID="24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd"
Sep 17 21:27:35.521290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.521052    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd"} err="failed to get container status \"24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd\": rpc error: code = NotFound desc = could not find container \"24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd\": container with ID starting with 24b4e713e4645e6089b394f2e74cb8ef1d2a3213ad2c3dd91232b80a945cfccd not found: ID does not exist"
Sep 17 21:27:35.521290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.521077    2586 scope.go:117] "RemoveContainer" containerID="d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d"
Sep 17 21:27:35.521589 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:27:35.521573    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d\": container with ID starting with d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d not found: ID does not exist" containerID="d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d"
Sep 17 21:27:35.521632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.521595    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d"} err="failed to get container status \"d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d\": rpc error: code = NotFound desc = could not find container \"d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d\": container with ID starting with d195abe84d73c149a6006218b3def2771539d02a01ac60e7ad0e5ad9ec26054d not found: ID does not exist"
Sep 17 21:27:35.521632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.521609    2586 scope.go:117] "RemoveContainer" containerID="58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc"
Sep 17 21:27:35.521875 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:27:35.521856    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc\": container with ID starting with 58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc not found: ID does not exist" containerID="58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc"
Sep 17 21:27:35.521926 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.521880    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc"} err="failed to get container status \"58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc\": rpc error: code = NotFound desc = could not find container \"58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc\": container with ID starting with 58328b780c06e0c05caa6719cf965409f92fb172757c55544d65aca93a8bb8bc not found: ID does not exist"
Sep 17 21:27:35.522725 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.522708    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"]
Sep 17 21:27:35.528883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.528860    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-4trbg"]
Sep 17 21:27:35.669888 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.669859    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"]
Sep 17 21:27:35.670136 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670125    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="storage-initializer"
Sep 17 21:27:35.670184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670139    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="storage-initializer"
Sep 17 21:27:35.670184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670147    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kube-rbac-proxy"
Sep 17 21:27:35.670184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670153    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kube-rbac-proxy"
Sep 17 21:27:35.670184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670160    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container"
Sep 17 21:27:35.670184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670166    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container"
Sep 17 21:27:35.670334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670207    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kube-rbac-proxy"
Sep 17 21:27:35.670334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.670217    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" containerName="kserve-container"
Sep 17 21:27:35.673583 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.673566    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.679976 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.679957    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 21:27:35.680082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.679987    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 21:27:35.687067 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.687040    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"]
Sep 17 21:27:35.759418 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.759334    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"]
Sep 17 21:27:35.762072 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.762056    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.765173 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.765156    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 21:27:35.776649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.776620    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"]
Sep 17 21:27:35.811056 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811030    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.811156 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811066    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.811156 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2eb254-8f59-4425-b5c8-70d70315b498-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.811156 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811136    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.811289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811166    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb4bz\" (UniqueName: \"kubernetes.io/projected/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-api-access-vb4bz\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.811289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811184    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2eb254-8f59-4425-b5c8-70d70315b498-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.811289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811208    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.811289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811227    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.811289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811271    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.811502 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.811317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvh7l\" (UniqueName: \"kubernetes.io/projected/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-api-access-jvh7l\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.911827 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2eb254-8f59-4425-b5c8-70d70315b498-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.911985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.911985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911862    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb4bz\" (UniqueName: \"kubernetes.io/projected/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-api-access-vb4bz\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.911985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2eb254-8f59-4425-b5c8-70d70315b498-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.911985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911917    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.911985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911939    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.911985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911967    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.912305 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.911999    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvh7l\" (UniqueName: \"kubernetes.io/projected/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-api-access-jvh7l\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.912305 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.912448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.912711 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912689    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.912746 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912689    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.912782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912769    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.912820 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912802    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.912857 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912827    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.912948 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.912929    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2eb254-8f59-4425-b5c8-70d70315b498-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.914932 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.914912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.915125 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.915102    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2eb254-8f59-4425-b5c8-70d70315b498-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.920071 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.920053    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvh7l\" (UniqueName: \"kubernetes.io/projected/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-api-access-jvh7l\") pod \"isvc-canary-stop-v2-predictor-d65d59944-wt2qg\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:35.920329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.920310    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb4bz\" (UniqueName: \"kubernetes.io/projected/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-api-access-vb4bz\") pod \"isvc-canary-stop-predictor-55656fdf77-rf9lp\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:35.982917 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:35.982888    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:36.071749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.071703    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:36.256285 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.256260    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"]
Sep 17 21:27:36.257943 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:27:36.257919    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb4cdd7_3bb8_4e8f_a6fb_63cf8d98844a.slice/crio-8bb1f527ac2faf49e30deb7a544c8d449df9c91894316c61bbc9ccdf3148ac87 WatchSource:0}: Error finding container 8bb1f527ac2faf49e30deb7a544c8d449df9c91894316c61bbc9ccdf3148ac87: Status 404 returned error can't find the container with id 8bb1f527ac2faf49e30deb7a544c8d449df9c91894316c61bbc9ccdf3148ac87
Sep 17 21:27:36.365221 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.365191    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"]
Sep 17 21:27:36.369561 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:27:36.369521    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2eb254_8f59_4425_b5c8_70d70315b498.slice/crio-a8d1f7aadba90b1657144aac6b873fb23d7902d31342f78e2045cd7fce821253 WatchSource:0}: Error finding container a8d1f7aadba90b1657144aac6b873fb23d7902d31342f78e2045cd7fce821253: Status 404 returned error can't find the container with id a8d1f7aadba90b1657144aac6b873fb23d7902d31342f78e2045cd7fce821253
Sep 17 21:27:36.501758 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.501718    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerStarted","Data":"2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296"}
Sep 17 21:27:36.502151 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.501780    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerStarted","Data":"a8d1f7aadba90b1657144aac6b873fb23d7902d31342f78e2045cd7fce821253"}
Sep 17 21:27:36.503492 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.503466    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerStarted","Data":"70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118"}
Sep 17 21:27:36.503608 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.503524    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerStarted","Data":"8bb1f527ac2faf49e30deb7a544c8d449df9c91894316c61bbc9ccdf3148ac87"}
Sep 17 21:27:36.671415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:36.671379    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e488f1bd-6e6c-4597-93d6-f65ede8da1c1" path="/var/lib/kubelet/pods/e488f1bd-6e6c-4597-93d6-f65ede8da1c1/volumes"
Sep 17 21:27:40.521190 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:40.521155    2586 generic.go:356] "Generic (PLEG): container finished" podID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerID="2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296" exitCode=0
Sep 17 21:27:40.521638 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:40.521200    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerDied","Data":"2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296"}
Sep 17 21:27:40.522947 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:40.522924    2586 generic.go:356] "Generic (PLEG): container finished" podID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerID="70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118" exitCode=0
Sep 17 21:27:40.523052 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:40.522983    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerDied","Data":"70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118"}
Sep 17 21:27:41.528265 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.528225    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerStarted","Data":"f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5"}
Sep 17 21:27:41.528751 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.528328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerStarted","Data":"7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac"}
Sep 17 21:27:41.528751 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.528658    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:41.528883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.528807    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:41.530275 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.530246    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:27:41.530397 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.530382    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerStarted","Data":"460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e"}
Sep 17 21:27:41.530438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.530423    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerStarted","Data":"72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c"}
Sep 17 21:27:41.530681 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.530665    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:41.530723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.530712    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:41.531540 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.531514    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:27:41.549118 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.549074    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podStartSLOduration=2.773051053 podStartE2EDuration="6.54906107s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.776010017s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:27:41.547305456 +0000 UTC m=+2277.081245266" watchObservedRunningTime="2026-09-17 21:27:41.54906107 +0000 UTC m=+2277.083000947"
Sep 17 21:27:41.567546 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:41.567510    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podStartSLOduration=2.872466221 podStartE2EDuration="6.567498379s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695032158s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:27:41.566633989 +0000 UTC m=+2277.100573795" watchObservedRunningTime="2026-09-17 21:27:41.567498379 +0000 UTC m=+2277.101438187"
Sep 17 21:27:42.534664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:42.534627    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:27:42.535067 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:42.534619    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:27:44.286505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:44.286478    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:27:47.538762 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:47.538731    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:27:47.539208 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:47.539172    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:27:47.539286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:47.539165    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:27:47.539527 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:47.539491    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:27:51.536526 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:51.536493    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"]
Sep 17 21:27:51.536925 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:51.536709    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" containerID="cri-o://9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a" gracePeriod=30
Sep 17 21:27:51.536925 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:51.536739    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kube-rbac-proxy" containerID="cri-o://0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829" gracePeriod=30
Sep 17 21:27:52.574931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:52.574882    2586 generic.go:356] "Generic (PLEG): container finished" podID="336baf50-870c-431d-8560-463815c21bba" containerID="0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829" exitCode=2
Sep 17 21:27:52.575295 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:52.574945    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerDied","Data":"0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829"}
Sep 17 21:27:54.281504 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:54.281460    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 21:27:54.286909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:54.286881    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 21:27:55.976600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:55.976577    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:27:56.066347 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066321    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336baf50-870c-431d-8560-463815c21bba-kserve-provision-location\") pod \"336baf50-870c-431d-8560-463815c21bba\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") "
Sep 17 21:27:56.066525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066387    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-kube-rbac-proxy-sar-config\") pod \"336baf50-870c-431d-8560-463815c21bba\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") "
Sep 17 21:27:56.066525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066410    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336baf50-870c-431d-8560-463815c21bba-proxy-tls\") pod \"336baf50-870c-431d-8560-463815c21bba\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") "
Sep 17 21:27:56.066525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066436    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm899\" (UniqueName: \"kubernetes.io/projected/336baf50-870c-431d-8560-463815c21bba-kube-api-access-mm899\") pod \"336baf50-870c-431d-8560-463815c21bba\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") "
Sep 17 21:27:56.066525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066475    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-cabundle-cert\") pod \"336baf50-870c-431d-8560-463815c21bba\" (UID: \"336baf50-870c-431d-8560-463815c21bba\") "
Sep 17 21:27:56.066755 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066630    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336baf50-870c-431d-8560-463815c21bba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "336baf50-870c-431d-8560-463815c21bba" (UID: "336baf50-870c-431d-8560-463815c21bba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:27:56.066814 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066795    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "336baf50-870c-431d-8560-463815c21bba" (UID: "336baf50-870c-431d-8560-463815c21bba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:27:56.066884 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.066862    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "336baf50-870c-431d-8560-463815c21bba" (UID: "336baf50-870c-431d-8560-463815c21bba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:27:56.068495 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.068449    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336baf50-870c-431d-8560-463815c21bba-kube-api-access-mm899" (OuterVolumeSpecName: "kube-api-access-mm899") pod "336baf50-870c-431d-8560-463815c21bba" (UID: "336baf50-870c-431d-8560-463815c21bba"). InnerVolumeSpecName "kube-api-access-mm899". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:27:56.068495 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.068486    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336baf50-870c-431d-8560-463815c21bba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "336baf50-870c-431d-8560-463815c21bba" (UID: "336baf50-870c-431d-8560-463815c21bba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:27:56.167093 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.167067    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:56.167093 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.167091    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/336baf50-870c-431d-8560-463815c21bba-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:56.167261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.167102    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/336baf50-870c-431d-8560-463815c21bba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:56.167261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.167112    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/336baf50-870c-431d-8560-463815c21bba-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:56.167261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.167121    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm899\" (UniqueName: \"kubernetes.io/projected/336baf50-870c-431d-8560-463815c21bba-kube-api-access-mm899\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:27:56.593262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.593220    2586 generic.go:356] "Generic (PLEG): container finished" podID="336baf50-870c-431d-8560-463815c21bba" containerID="9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a" exitCode=0
Sep 17 21:27:56.593450 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.593282    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"
Sep 17 21:27:56.593450 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.593293    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerDied","Data":"9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a"}
Sep 17 21:27:56.593450 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.593376    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9" event={"ID":"336baf50-870c-431d-8560-463815c21bba","Type":"ContainerDied","Data":"61f495ad7fdf17d2473cdbf61fe0b475f4f55e608783834b61cd2d0b7700fb03"}
Sep 17 21:27:56.593450 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.593409    2586 scope.go:117] "RemoveContainer" containerID="0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829"
Sep 17 21:27:56.602701 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.602687    2586 scope.go:117] "RemoveContainer" containerID="9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a"
Sep 17 21:27:56.609943 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.609925    2586 scope.go:117] "RemoveContainer" containerID="5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27"
Sep 17 21:27:56.617681 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.617657    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"]
Sep 17 21:27:56.618440 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.618422    2586 scope.go:117] "RemoveContainer" containerID="0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829"
Sep 17 21:27:56.618747 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:27:56.618722    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829\": container with ID starting with 0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829 not found: ID does not exist" containerID="0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829"
Sep 17 21:27:56.618856 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.618752    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829"} err="failed to get container status \"0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829\": rpc error: code = NotFound desc = could not find container \"0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829\": container with ID starting with 0cc5dba4ce45e8c11700233b10192b2b05c7074f05709cf49650f132b350f829 not found: ID does not exist"
Sep 17 21:27:56.618856 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.618772    2586 scope.go:117] "RemoveContainer" containerID="9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a"
Sep 17 21:27:56.619051 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:27:56.619034    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a\": container with ID starting with 9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a not found: ID does not exist" containerID="9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a"
Sep 17 21:27:56.619114 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.619053    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a"} err="failed to get container status \"9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a\": rpc error: code = NotFound desc = could not find container \"9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a\": container with ID starting with 9d37b6530fa2bf4c4508120ebff25873e724fa9b436bbdd6cf5074be3026d13a not found: ID does not exist"
Sep 17 21:27:56.619114 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.619066    2586 scope.go:117] "RemoveContainer" containerID="5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27"
Sep 17 21:27:56.619291 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:27:56.619272    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27\": container with ID starting with 5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27 not found: ID does not exist" containerID="5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27"
Sep 17 21:27:56.619468 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.619292    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27"} err="failed to get container status \"5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27\": rpc error: code = NotFound desc = could not find container \"5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27\": container with ID starting with 5914b2236264108df38ff683685898b9d933cf195f115c92945e9b0f73033b27 not found: ID does not exist"
Sep 17 21:27:56.626651 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.626631    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-ldld9"]
Sep 17 21:27:56.671767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.671743    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336baf50-870c-431d-8560-463815c21bba" path="/var/lib/kubelet/pods/336baf50-870c-431d-8560-463815c21bba/volumes"
Sep 17 21:27:56.855720 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.855647    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"]
Sep 17 21:27:56.855927 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.855915    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kube-rbac-proxy"
Sep 17 21:27:56.855975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.855929    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kube-rbac-proxy"
Sep 17 21:27:56.855975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.855947    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="storage-initializer"
Sep 17 21:27:56.855975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.855952    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="storage-initializer"
Sep 17 21:27:56.855975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.855958    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container"
Sep 17 21:27:56.855975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.855963    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container"
Sep 17 21:27:56.856126 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.856019    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kserve-container"
Sep 17 21:27:56.856126 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.856042    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="336baf50-870c-431d-8560-463815c21bba" containerName="kube-rbac-proxy"
Sep 17 21:27:56.859795 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.859778    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:56.868444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.868424    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 21:27:56.868597 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.868580    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 21:27:56.884290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.884267    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"]
Sep 17 21:27:56.971271 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.971239    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4622e3b-9c9c-4881-a556-3eb7890684cb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:56.971456 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.971290    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:56.971456 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.971317    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4622e3b-9c9c-4881-a556-3eb7890684cb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:56.971456 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.971344    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7kc\" (UniqueName: \"kubernetes.io/projected/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-api-access-wd7kc\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:56.971456 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:56.971381    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.072529 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.072496    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4622e3b-9c9c-4881-a556-3eb7890684cb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.072990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.072540    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.072990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.072557    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4622e3b-9c9c-4881-a556-3eb7890684cb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.072990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.072575    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7kc\" (UniqueName: \"kubernetes.io/projected/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-api-access-wd7kc\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.072990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.072701    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.073226 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.073088    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4622e3b-9c9c-4881-a556-3eb7890684cb-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.073226 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.073208    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.073416 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.073395    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.074975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.074956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4622e3b-9c9c-4881-a556-3eb7890684cb-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.081008 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.080985    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7kc\" (UniqueName: \"kubernetes.io/projected/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-api-access-wd7kc\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.169820 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.169790    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:27:57.441447 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.441423    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"]
Sep 17 21:27:57.443612 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:27:57.443579    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4622e3b_9c9c_4881_a556_3eb7890684cb.slice/crio-7497d90778efde738ac553254a0950c86677a86b3e1f6d211aa91d05681b1721 WatchSource:0}: Error finding container 7497d90778efde738ac553254a0950c86677a86b3e1f6d211aa91d05681b1721: Status 404 returned error can't find the container with id 7497d90778efde738ac553254a0950c86677a86b3e1f6d211aa91d05681b1721
Sep 17 21:27:57.539877 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.539839    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:27:57.540005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.539868    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:27:57.599703 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.599675    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerStarted","Data":"9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1"}
Sep 17 21:27:57.599841 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:27:57.599739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerStarted","Data":"7497d90778efde738ac553254a0950c86677a86b3e1f6d211aa91d05681b1721"}
Sep 17 21:28:01.620056 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:01.620023    2586 generic.go:356] "Generic (PLEG): container finished" podID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerID="9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1" exitCode=0
Sep 17 21:28:01.620502 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:01.620106    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerDied","Data":"9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1"}
Sep 17 21:28:02.626736 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:02.626705    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerStarted","Data":"d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8"}
Sep 17 21:28:02.627210 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:02.626776    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerStarted","Data":"eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f"}
Sep 17 21:28:02.627210 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:02.626985    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:28:02.651547 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:02.651499    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" podStartSLOduration=2.982347447 podStartE2EDuration="6.651484245s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.669136798s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:28:02.649185764 +0000 UTC m=+2298.183125575" watchObservedRunningTime="2026-09-17 21:28:02.651484245 +0000 UTC m=+2298.185424054"
Sep 17 21:28:03.630392 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:03.630350    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:28:07.539575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:07.539534    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:28:07.540040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:07.539534    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:28:09.639734 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:09.639706    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:28:17.539838 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:17.539801    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:28:17.540222 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:17.539813    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:28:27.539754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:27.539670    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:28:27.539754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:27.539670    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:28:37.539700 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:37.539666    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:28:37.539700 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:37.539692    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:28:39.643656 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:39.643627    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:28:46.865909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:46.865877    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"]
Sep 17 21:28:46.866271 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:46.866115    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kserve-container" containerID="cri-o://eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f" gracePeriod=30
Sep 17 21:28:46.866271 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:46.866199    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kube-rbac-proxy" containerID="cri-o://d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8" gracePeriod=30
Sep 17 21:28:47.539220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:47.539184    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:28:47.539594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:47.539567    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:28:47.794416 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:47.794327    2586 generic.go:356] "Generic (PLEG): container finished" podID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerID="d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8" exitCode=2
Sep 17 21:28:47.794416 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:47.794401    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerDied","Data":"d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8"}
Sep 17 21:28:49.634864 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:49.634825    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 21:28:49.640267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:49.640245    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 21:28:52.119650 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.119603    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"]
Sep 17 21:28:52.122794 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.122776    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.127629 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.127607    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 21:28:52.127747 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.127683    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 21:28:52.141438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.141412    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"]
Sep 17 21:28:52.174992 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.174967    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.175100 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.174999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348b8f0f-d706-43f5-b022-16c572c5c06e-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.175100 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.175021    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2x7t\" (UniqueName: \"kubernetes.io/projected/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-api-access-r2x7t\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.175183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.175111    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.175183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.175154    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348b8f0f-d706-43f5-b022-16c572c5c06e-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.275986 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.275949    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.275986 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.275994    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348b8f0f-d706-43f5-b022-16c572c5c06e-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.276243 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.276132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.276243 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.276192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348b8f0f-d706-43f5-b022-16c572c5c06e-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.276243 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.276223    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2x7t\" (UniqueName: \"kubernetes.io/projected/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-api-access-r2x7t\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.276449 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.276401    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348b8f0f-d706-43f5-b022-16c572c5c06e-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.276750 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.276725    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.276835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.276730    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.278673 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.278654    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348b8f0f-d706-43f5-b022-16c572c5c06e-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.286746 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.286719    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2x7t\" (UniqueName: \"kubernetes.io/projected/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-api-access-r2x7t\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.432494 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.432414    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:52.711393 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.711257    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"]
Sep 17 21:28:52.714070 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:28:52.714027    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348b8f0f_d706_43f5_b022_16c572c5c06e.slice/crio-b05c985c98fdcf90eec616aefe9665af1e39d32243b56f8a316f09376703de1a WatchSource:0}: Error finding container b05c985c98fdcf90eec616aefe9665af1e39d32243b56f8a316f09376703de1a: Status 404 returned error can't find the container with id b05c985c98fdcf90eec616aefe9665af1e39d32243b56f8a316f09376703de1a
Sep 17 21:28:52.815954 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.815919    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerStarted","Data":"ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888"}
Sep 17 21:28:52.816119 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:52.815989    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerStarted","Data":"b05c985c98fdcf90eec616aefe9665af1e39d32243b56f8a316f09376703de1a"}
Sep 17 21:28:53.188082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.188061    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:28:53.285368 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285271    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4622e3b-9c9c-4881-a556-3eb7890684cb-proxy-tls\") pod \"b4622e3b-9c9c-4881-a556-3eb7890684cb\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") "
Sep 17 21:28:53.285368 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285324    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd7kc\" (UniqueName: \"kubernetes.io/projected/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-api-access-wd7kc\") pod \"b4622e3b-9c9c-4881-a556-3eb7890684cb\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") "
Sep 17 21:28:53.285589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285370    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-rbac-proxy-sar-config\") pod \"b4622e3b-9c9c-4881-a556-3eb7890684cb\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") "
Sep 17 21:28:53.285589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285407    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-cabundle-cert\") pod \"b4622e3b-9c9c-4881-a556-3eb7890684cb\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") "
Sep 17 21:28:53.285767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285596    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4622e3b-9c9c-4881-a556-3eb7890684cb-kserve-provision-location\") pod \"b4622e3b-9c9c-4881-a556-3eb7890684cb\" (UID: \"b4622e3b-9c9c-4881-a556-3eb7890684cb\") "
Sep 17 21:28:53.286058 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285773    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4622e3b-9c9c-4881-a556-3eb7890684cb" (UID: "b4622e3b-9c9c-4881-a556-3eb7890684cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:28:53.286058 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285924    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4622e3b-9c9c-4881-a556-3eb7890684cb" (UID: "b4622e3b-9c9c-4881-a556-3eb7890684cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:28:53.286058 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.285962    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4622e3b-9c9c-4881-a556-3eb7890684cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4622e3b-9c9c-4881-a556-3eb7890684cb" (UID: "b4622e3b-9c9c-4881-a556-3eb7890684cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:28:53.287975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.287944    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4622e3b-9c9c-4881-a556-3eb7890684cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4622e3b-9c9c-4881-a556-3eb7890684cb" (UID: "b4622e3b-9c9c-4881-a556-3eb7890684cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:28:53.288073 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.287976    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-api-access-wd7kc" (OuterVolumeSpecName: "kube-api-access-wd7kc") pod "b4622e3b-9c9c-4881-a556-3eb7890684cb" (UID: "b4622e3b-9c9c-4881-a556-3eb7890684cb"). InnerVolumeSpecName "kube-api-access-wd7kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:28:53.386493 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.386460    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd7kc\" (UniqueName: \"kubernetes.io/projected/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-api-access-wd7kc\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:28:53.386493 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.386491    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:28:53.386689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.386504    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4622e3b-9c9c-4881-a556-3eb7890684cb-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:28:53.386689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.386516    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4622e3b-9c9c-4881-a556-3eb7890684cb-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:28:53.386689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.386529    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4622e3b-9c9c-4881-a556-3eb7890684cb-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:28:53.822283 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.822244    2586 generic.go:356] "Generic (PLEG): container finished" podID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerID="eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f" exitCode=0
Sep 17 21:28:53.822496 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.822310    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"
Sep 17 21:28:53.822496 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.822323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerDied","Data":"eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f"}
Sep 17 21:28:53.822496 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.822411    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk" event={"ID":"b4622e3b-9c9c-4881-a556-3eb7890684cb","Type":"ContainerDied","Data":"7497d90778efde738ac553254a0950c86677a86b3e1f6d211aa91d05681b1721"}
Sep 17 21:28:53.822496 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.822449    2586 scope.go:117] "RemoveContainer" containerID="d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8"
Sep 17 21:28:53.831100 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.831080    2586 scope.go:117] "RemoveContainer" containerID="eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f"
Sep 17 21:28:53.838155 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.838140    2586 scope.go:117] "RemoveContainer" containerID="9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1"
Sep 17 21:28:53.845435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.845417    2586 scope.go:117] "RemoveContainer" containerID="d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8"
Sep 17 21:28:53.845725 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:28:53.845707    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8\": container with ID starting with d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8 not found: ID does not exist" containerID="d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8"
Sep 17 21:28:53.845815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.845729    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8"} err="failed to get container status \"d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8\": rpc error: code = NotFound desc = could not find container \"d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8\": container with ID starting with d1ae0c29518239c89038abc7d21b7a60ff938ab49e636bf49a931331518b54d8 not found: ID does not exist"
Sep 17 21:28:53.845815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.845745    2586 scope.go:117] "RemoveContainer" containerID="eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f"
Sep 17 21:28:53.845989 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:28:53.845974    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f\": container with ID starting with eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f not found: ID does not exist" containerID="eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f"
Sep 17 21:28:53.846049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.845992    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f"} err="failed to get container status \"eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f\": rpc error: code = NotFound desc = could not find container \"eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f\": container with ID starting with eed129a5d85c2a4b8b5696c06d5dc9f499dca8099750a18f5e38ef561663a94f not found: ID does not exist"
Sep 17 21:28:53.846049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.846021    2586 scope.go:117] "RemoveContainer" containerID="9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1"
Sep 17 21:28:53.846264 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:28:53.846249    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1\": container with ID starting with 9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1 not found: ID does not exist" containerID="9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1"
Sep 17 21:28:53.846328 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.846267    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1"} err="failed to get container status \"9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1\": rpc error: code = NotFound desc = could not find container \"9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1\": container with ID starting with 9a0c46f539c3054465ddc29380c483978eff841f855dfd26593aba99a8eeeae1 not found: ID does not exist"
Sep 17 21:28:53.856711 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.856688    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"]
Sep 17 21:28:53.862103 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:53.862083    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zgrdk"]
Sep 17 21:28:54.670993 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:54.670962    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" path="/var/lib/kubelet/pods/b4622e3b-9c9c-4881-a556-3eb7890684cb/volumes"
Sep 17 21:28:56.839863 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:56.839781    2586 generic.go:356] "Generic (PLEG): container finished" podID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerID="ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888" exitCode=0
Sep 17 21:28:56.839863 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:56.839826    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerDied","Data":"ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888"}
Sep 17 21:28:57.539804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:57.539773    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:28:57.540508 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:57.540486    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:28:57.846041 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:57.845965    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerStarted","Data":"a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28"}
Sep 17 21:28:57.846041 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:57.846029    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerStarted","Data":"7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186"}
Sep 17 21:28:57.846447 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:57.846238    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:28:57.869962 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:57.869914    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" podStartSLOduration=2.158315118 podStartE2EDuration="5.869901428s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.71158631s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:28:57.868161913 +0000 UTC m=+2353.402101749" watchObservedRunningTime="2026-09-17 21:28:57.869901428 +0000 UTC m=+2353.403841284"
Sep 17 21:28:58.849698 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:28:58.849670    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:29:04.859024 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:04.858996    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:29:05.679973 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.679943    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"]
Sep 17 21:29:05.680225 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.680193    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" containerID="cri-o://7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac" gracePeriod=30
Sep 17 21:29:05.680489 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.680209    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kube-rbac-proxy" containerID="cri-o://f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5" gracePeriod=30
Sep 17 21:29:05.741177 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.741150    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"]
Sep 17 21:29:05.741417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.741392    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" containerID="cri-o://72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c" gracePeriod=30
Sep 17 21:29:05.741519 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.741428    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kube-rbac-proxy" containerID="cri-o://460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e" gracePeriod=30
Sep 17 21:29:05.876778 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.876753    2586 generic.go:356] "Generic (PLEG): container finished" podID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerID="460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e" exitCode=2
Sep 17 21:29:05.877119 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.876825    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerDied","Data":"460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e"}
Sep 17 21:29:05.878768 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.878747    2586 generic.go:356] "Generic (PLEG): container finished" podID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerID="f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5" exitCode=2
Sep 17 21:29:05.878847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:05.878781    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerDied","Data":"f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5"}
Sep 17 21:29:07.535257 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:07.535219    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 21:29:07.535643 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:07.535219    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 21:29:07.539639 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:07.539620    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 21:29:07.539723 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:07.539649    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 21:29:10.394350 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.394325    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:29:10.512183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512150    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb4bz\" (UniqueName: \"kubernetes.io/projected/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-api-access-vb4bz\") pod \"6f2eb254-8f59-4425-b5c8-70d70315b498\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") "
Sep 17 21:29:10.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512221    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2eb254-8f59-4425-b5c8-70d70315b498-proxy-tls\") pod \"6f2eb254-8f59-4425-b5c8-70d70315b498\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") "
Sep 17 21:29:10.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512248    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2eb254-8f59-4425-b5c8-70d70315b498-kserve-provision-location\") pod \"6f2eb254-8f59-4425-b5c8-70d70315b498\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") "
Sep 17 21:29:10.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512288    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-cabundle-cert\") pod \"6f2eb254-8f59-4425-b5c8-70d70315b498\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") "
Sep 17 21:29:10.512340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512316    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-rbac-proxy-sar-config\") pod \"6f2eb254-8f59-4425-b5c8-70d70315b498\" (UID: \"6f2eb254-8f59-4425-b5c8-70d70315b498\") "
Sep 17 21:29:10.512673 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512640    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2eb254-8f59-4425-b5c8-70d70315b498-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f2eb254-8f59-4425-b5c8-70d70315b498" (UID: "6f2eb254-8f59-4425-b5c8-70d70315b498"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:29:10.512742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512669    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f2eb254-8f59-4425-b5c8-70d70315b498" (UID: "6f2eb254-8f59-4425-b5c8-70d70315b498"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:10.512742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.512705    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f2eb254-8f59-4425-b5c8-70d70315b498" (UID: "6f2eb254-8f59-4425-b5c8-70d70315b498"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:10.514280 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.514260    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-api-access-vb4bz" (OuterVolumeSpecName: "kube-api-access-vb4bz") pod "6f2eb254-8f59-4425-b5c8-70d70315b498" (UID: "6f2eb254-8f59-4425-b5c8-70d70315b498"). InnerVolumeSpecName "kube-api-access-vb4bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:29:10.514778 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.514759    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2eb254-8f59-4425-b5c8-70d70315b498-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f2eb254-8f59-4425-b5c8-70d70315b498" (UID: "6f2eb254-8f59-4425-b5c8-70d70315b498"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:29:10.613267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.613199    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb4bz\" (UniqueName: \"kubernetes.io/projected/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-api-access-vb4bz\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.613267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.613231    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f2eb254-8f59-4425-b5c8-70d70315b498-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.613267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.613247    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f2eb254-8f59-4425-b5c8-70d70315b498-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.613267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.613260    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.613590 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.613275    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f2eb254-8f59-4425-b5c8-70d70315b498-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.721672 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.721650    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:29:10.814742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.814714    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-rbac-proxy-sar-config\") pod \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") "
Sep 17 21:29:10.814886 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.814758    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-proxy-tls\") pod \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") "
Sep 17 21:29:10.814886 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.814788    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kserve-provision-location\") pod \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") "
Sep 17 21:29:10.814886 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.814846    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-cabundle-cert\") pod \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") "
Sep 17 21:29:10.814886 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.814885    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvh7l\" (UniqueName: \"kubernetes.io/projected/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-api-access-jvh7l\") pod \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\" (UID: \"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a\") "
Sep 17 21:29:10.815198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.815164    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" (UID: "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:29:10.815198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.815186    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" (UID: "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:10.815400 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.815290    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" (UID: "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:10.816821 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.816798    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" (UID: "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:29:10.816905 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.816862    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-api-access-jvh7l" (OuterVolumeSpecName: "kube-api-access-jvh7l") pod "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" (UID: "eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a"). InnerVolumeSpecName "kube-api-access-jvh7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:29:10.908053 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.907978    2586 generic.go:356] "Generic (PLEG): container finished" podID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerID="72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c" exitCode=0
Sep 17 21:29:10.908182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.908055    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"
Sep 17 21:29:10.908182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.908056    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerDied","Data":"72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c"}
Sep 17 21:29:10.908182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.908153    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp" event={"ID":"6f2eb254-8f59-4425-b5c8-70d70315b498","Type":"ContainerDied","Data":"a8d1f7aadba90b1657144aac6b873fb23d7902d31342f78e2045cd7fce821253"}
Sep 17 21:29:10.908376 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.908194    2586 scope.go:117] "RemoveContainer" containerID="460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e"
Sep 17 21:29:10.910411 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.910389    2586 generic.go:356] "Generic (PLEG): container finished" podID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerID="7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac" exitCode=0
Sep 17 21:29:10.910529 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.910471    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerDied","Data":"7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac"}
Sep 17 21:29:10.910605 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.910534    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"
Sep 17 21:29:10.910605 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.910541    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg" event={"ID":"eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a","Type":"ContainerDied","Data":"8bb1f527ac2faf49e30deb7a544c8d449df9c91894316c61bbc9ccdf3148ac87"}
Sep 17 21:29:10.915269 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.915250    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.915386 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.915271    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.915386 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.915285    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.915386 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.915297    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvh7l\" (UniqueName: \"kubernetes.io/projected/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-api-access-jvh7l\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.915386 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.915310    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:10.918206 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.918187    2586 scope.go:117] "RemoveContainer" containerID="72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c"
Sep 17 21:29:10.925461 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.925436    2586 scope.go:117] "RemoveContainer" containerID="2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296"
Sep 17 21:29:10.929264 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.928963    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"]
Sep 17 21:29:10.930514 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.930496    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-rf9lp"]
Sep 17 21:29:10.932959 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.932943    2586 scope.go:117] "RemoveContainer" containerID="460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e"
Sep 17 21:29:10.933211 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:10.933193    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e\": container with ID starting with 460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e not found: ID does not exist" containerID="460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e"
Sep 17 21:29:10.933261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.933218    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e"} err="failed to get container status \"460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e\": rpc error: code = NotFound desc = could not find container \"460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e\": container with ID starting with 460667f61c89a093a591268b2073c7ed9a577c37c76e4b7944b3beb56e8d325e not found: ID does not exist"
Sep 17 21:29:10.933261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.933235    2586 scope.go:117] "RemoveContainer" containerID="72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c"
Sep 17 21:29:10.933477 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:10.933464    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c\": container with ID starting with 72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c not found: ID does not exist" containerID="72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c"
Sep 17 21:29:10.933527 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.933480    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c"} err="failed to get container status \"72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c\": rpc error: code = NotFound desc = could not find container \"72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c\": container with ID starting with 72c5ccf359f6d1492cfbac0b4fcbae3aa53f2a0c4518b84e46208cd206fe6d9c not found: ID does not exist"
Sep 17 21:29:10.933527 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.933491    2586 scope.go:117] "RemoveContainer" containerID="2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296"
Sep 17 21:29:10.933754 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:10.933736    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296\": container with ID starting with 2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296 not found: ID does not exist" containerID="2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296"
Sep 17 21:29:10.933803 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.933758    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296"} err="failed to get container status \"2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296\": rpc error: code = NotFound desc = could not find container \"2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296\": container with ID starting with 2141c36b64e493d7738dce4f119866372a7208b17d7c2aa8af9135ae4ab54296 not found: ID does not exist"
Sep 17 21:29:10.933803 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.933772    2586 scope.go:117] "RemoveContainer" containerID="f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5"
Sep 17 21:29:10.940619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.940603    2586 scope.go:117] "RemoveContainer" containerID="7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac"
Sep 17 21:29:10.947499 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.947481    2586 scope.go:117] "RemoveContainer" containerID="70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118"
Sep 17 21:29:10.947675 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.947661    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"]
Sep 17 21:29:10.952755 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.952736    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-wt2qg"]
Sep 17 21:29:10.954389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.954375    2586 scope.go:117] "RemoveContainer" containerID="f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5"
Sep 17 21:29:10.954651 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:10.954633    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5\": container with ID starting with f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5 not found: ID does not exist" containerID="f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5"
Sep 17 21:29:10.954713 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.954657    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5"} err="failed to get container status \"f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5\": rpc error: code = NotFound desc = could not find container \"f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5\": container with ID starting with f1a27df1c66cd6b643c9f84cf7048563452c63794a7f64935ad2d4bc8e9677d5 not found: ID does not exist"
Sep 17 21:29:10.954713 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.954682    2586 scope.go:117] "RemoveContainer" containerID="7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac"
Sep 17 21:29:10.954936 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:10.954920    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac\": container with ID starting with 7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac not found: ID does not exist" containerID="7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac"
Sep 17 21:29:10.954979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.954942    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac"} err="failed to get container status \"7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac\": rpc error: code = NotFound desc = could not find container \"7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac\": container with ID starting with 7d0c5e98ea0d2e3156e22713d2a00496164886ff1258b72f5c35fdf8fe52b4ac not found: ID does not exist"
Sep 17 21:29:10.954979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.954955    2586 scope.go:117] "RemoveContainer" containerID="70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118"
Sep 17 21:29:10.955188 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:10.955174    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118\": container with ID starting with 70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118 not found: ID does not exist" containerID="70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118"
Sep 17 21:29:10.955237 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:10.955190    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118"} err="failed to get container status \"70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118\": rpc error: code = NotFound desc = could not find container \"70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118\": container with ID starting with 70038fa94ddd402c26fb86b253b303174b2ba7b1482fd186a7bfeee2a8a2b118 not found: ID does not exist"
Sep 17 21:29:12.670733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:12.670699    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" path="/var/lib/kubelet/pods/6f2eb254-8f59-4425-b5c8-70d70315b498/volumes"
Sep 17 21:29:12.671206 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:12.671191    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" path="/var/lib/kubelet/pods/eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a/volumes"
Sep 17 21:29:15.926683 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.926647    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"]
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927002    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="storage-initializer"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927019    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="storage-initializer"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927037    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="storage-initializer"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927045    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="storage-initializer"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927054    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927063    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927077    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927085    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927096    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927104    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927115    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927123    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927130    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927138    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927147    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kserve-container"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927155    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kserve-container"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927165    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="storage-initializer"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927174    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="storage-initializer"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927243    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kserve-container"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927256    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kserve-container"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927265    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kserve-container"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927273    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2eb254-8f59-4425-b5c8-70d70315b498" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927284    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb4cdd7-3bb8-4e8f-a6fb-63cf8d98844a" containerName="kube-rbac-proxy"
Sep 17 21:29:15.927843 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.927293    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4622e3b-9c9c-4881-a556-3eb7890684cb" containerName="kube-rbac-proxy"
Sep 17 21:29:15.930485 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.930465    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:15.933010 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.932989    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 21:29:15.933106 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.933033    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 21:29:15.942174 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:15.942155    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"]
Sep 17 21:29:16.053077 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.053046    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.053224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.053085    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.053224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.053107    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.053224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.053166    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww5x\" (UniqueName: \"kubernetes.io/projected/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-api-access-mww5x\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.053224 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.053205    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.153818 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.153783    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.153978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.153823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww5x\" (UniqueName: \"kubernetes.io/projected/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-api-access-mww5x\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.153978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.153852    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.153978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.153910    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.153978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.153940    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.154332 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.154306    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.154526 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.154507    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.154599 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.154578    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.156329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.156312    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.162471 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.162451    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww5x\" (UniqueName: \"kubernetes.io/projected/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-api-access-mww5x\") pod \"isvc-lightgbm-predictor-54f85ddf88-7gfn4\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.240498 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.240418    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:16.517773 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.517746    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"]
Sep 17 21:29:16.519414 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:29:16.519386    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a96751_7842_4ddd_9cc3_a59d2ebae7c7.slice/crio-ca2921a47fc2a372b3f57bb37f2e499204e3ce1688b8e826edaa60f2b5926660 WatchSource:0}: Error finding container ca2921a47fc2a372b3f57bb37f2e499204e3ce1688b8e826edaa60f2b5926660: Status 404 returned error can't find the container with id ca2921a47fc2a372b3f57bb37f2e499204e3ce1688b8e826edaa60f2b5926660
Sep 17 21:29:16.936245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.936200    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerStarted","Data":"e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6"}
Sep 17 21:29:16.936733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:16.936278    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerStarted","Data":"ca2921a47fc2a372b3f57bb37f2e499204e3ce1688b8e826edaa60f2b5926660"}
Sep 17 21:29:20.951941 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:20.951909    2586 generic.go:356] "Generic (PLEG): container finished" podID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerID="e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6" exitCode=0
Sep 17 21:29:20.952326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:20.951983    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerDied","Data":"e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6"}
Sep 17 21:29:20.953232 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:20.953215    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:29:27.983201 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:27.983165    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerStarted","Data":"93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7"}
Sep 17 21:29:27.983595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:27.983233    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerStarted","Data":"fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181"}
Sep 17 21:29:27.983595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:27.983423    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:28.004041 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:28.003995    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podStartSLOduration=3.2735235879999998 podStartE2EDuration="13.003977104s" totalImagesPullingTime="6.083879386s" totalInitContainerRuntime="3.64657413s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:29:28.002866091 +0000 UTC m=+2383.536805899" watchObservedRunningTime="2026-09-17 21:29:28.003977104 +0000 UTC m=+2383.537916912"
Sep 17 21:29:28.986786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:28.986748    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:28.988148 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:28.988123    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:29:29.990291 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:29.990248    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:29:34.862915 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:34.862882    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:29:34.995194 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:34.995167    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:29:34.995627 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:34.995602    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:29:42.100086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:42.100047    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"]
Sep 17 21:29:42.100632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:42.100313    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kserve-container" containerID="cri-o://7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186" gracePeriod=30
Sep 17 21:29:42.100632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:42.100401    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kube-rbac-proxy" containerID="cri-o://a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28" gracePeriod=30
Sep 17 21:29:43.036914 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:43.036883    2586 generic.go:356] "Generic (PLEG): container finished" podID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerID="a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28" exitCode=2
Sep 17 21:29:43.037082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:43.036939    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerDied","Data":"a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28"}
Sep 17 21:29:44.854473 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:44.854422    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 21:29:44.995998 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:44.995959    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:29:45.901523 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:45.901482    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.49:8080/v2/models/xgboost-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 21:29:47.407312 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.407278    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"]
Sep 17 21:29:47.410753 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.410732    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.415150 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.415132    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:29:47.415582 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.415559    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 21:29:47.428226 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.428203    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"]
Sep 17 21:29:47.589802 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.589771    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl6k6\" (UniqueName: \"kubernetes.io/projected/453681f5-8078-41b3-a4a1-12769d88ef54-kube-api-access-dl6k6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.589968 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.589808    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453681f5-8078-41b3-a4a1-12769d88ef54-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.589968 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.589834    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.589968 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.589893    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.589968 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.589942    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.690946 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.690876    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl6k6\" (UniqueName: \"kubernetes.io/projected/453681f5-8078-41b3-a4a1-12769d88ef54-kube-api-access-dl6k6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.690946 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.690917    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453681f5-8078-41b3-a4a1-12769d88ef54-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.691083 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.690948    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.691083 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.690973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.691083 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.691003    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.691226 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:47.691097    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 21:29:47.691226 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:47.691173    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls podName:453681f5-8078-41b3-a4a1-12769d88ef54 nodeName:}" failed. No retries permitted until 2026-09-17 21:29:48.191151332 +0000 UTC m=+2403.725091119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" (UID: "453681f5-8078-41b3-a4a1-12769d88ef54") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 21:29:47.691339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.691323    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453681f5-8078-41b3-a4a1-12769d88ef54-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.691629 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.691611    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.691713 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.691696    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:47.701076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:47.701052    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl6k6\" (UniqueName: \"kubernetes.io/projected/453681f5-8078-41b3-a4a1-12769d88ef54-kube-api-access-dl6k6\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:48.194485 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:48.194456    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:48.196951 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:48.196928    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-pcj2l\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:48.322561 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:48.322534    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:48.595442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:48.595251    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"]
Sep 17 21:29:48.597652 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:29:48.597626    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453681f5_8078_41b3_a4a1_12769d88ef54.slice/crio-622c22503162d6c056a6d25099797000c67a9ec546fc65459001cdee4cc1964e WatchSource:0}: Error finding container 622c22503162d6c056a6d25099797000c67a9ec546fc65459001cdee4cc1964e: Status 404 returned error can't find the container with id 622c22503162d6c056a6d25099797000c67a9ec546fc65459001cdee4cc1964e
Sep 17 21:29:49.060810 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:49.060769    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerStarted","Data":"a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b"}
Sep 17 21:29:49.060990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:49.060835    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerStarted","Data":"622c22503162d6c056a6d25099797000c67a9ec546fc65459001cdee4cc1964e"}
Sep 17 21:29:49.854591 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:49.854553    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 21:29:51.938596 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:51.938571    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:29:52.021661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.021586    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2x7t\" (UniqueName: \"kubernetes.io/projected/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-api-access-r2x7t\") pod \"348b8f0f-d706-43f5-b022-16c572c5c06e\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") "
Sep 17 21:29:52.021661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.021626    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-cabundle-cert\") pod \"348b8f0f-d706-43f5-b022-16c572c5c06e\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") "
Sep 17 21:29:52.021661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.021655    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348b8f0f-d706-43f5-b022-16c572c5c06e-proxy-tls\") pod \"348b8f0f-d706-43f5-b022-16c572c5c06e\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") "
Sep 17 21:29:52.021916 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.021684    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348b8f0f-d706-43f5-b022-16c572c5c06e-kserve-provision-location\") pod \"348b8f0f-d706-43f5-b022-16c572c5c06e\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") "
Sep 17 21:29:52.021916 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.021740    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-rbac-proxy-sar-config\") pod \"348b8f0f-d706-43f5-b022-16c572c5c06e\" (UID: \"348b8f0f-d706-43f5-b022-16c572c5c06e\") "
Sep 17 21:29:52.022049 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.022023    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348b8f0f-d706-43f5-b022-16c572c5c06e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "348b8f0f-d706-43f5-b022-16c572c5c06e" (UID: "348b8f0f-d706-43f5-b022-16c572c5c06e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:29:52.022193 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.022168    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "348b8f0f-d706-43f5-b022-16c572c5c06e" (UID: "348b8f0f-d706-43f5-b022-16c572c5c06e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:52.022248 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.022173    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "348b8f0f-d706-43f5-b022-16c572c5c06e" (UID: "348b8f0f-d706-43f5-b022-16c572c5c06e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:29:52.023718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.023694    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348b8f0f-d706-43f5-b022-16c572c5c06e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "348b8f0f-d706-43f5-b022-16c572c5c06e" (UID: "348b8f0f-d706-43f5-b022-16c572c5c06e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:29:52.023806 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.023789    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-api-access-r2x7t" (OuterVolumeSpecName: "kube-api-access-r2x7t") pod "348b8f0f-d706-43f5-b022-16c572c5c06e" (UID: "348b8f0f-d706-43f5-b022-16c572c5c06e"). InnerVolumeSpecName "kube-api-access-r2x7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:29:52.073819 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.073795    2586 generic.go:356] "Generic (PLEG): container finished" podID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerID="7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186" exitCode=0
Sep 17 21:29:52.073908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.073829    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerDied","Data":"7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186"}
Sep 17 21:29:52.073908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.073851    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"
Sep 17 21:29:52.073908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.073871    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd" event={"ID":"348b8f0f-d706-43f5-b022-16c572c5c06e","Type":"ContainerDied","Data":"b05c985c98fdcf90eec616aefe9665af1e39d32243b56f8a316f09376703de1a"}
Sep 17 21:29:52.073908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.073896    2586 scope.go:117] "RemoveContainer" containerID="a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28"
Sep 17 21:29:52.083302 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.083283    2586 scope.go:117] "RemoveContainer" containerID="7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186"
Sep 17 21:29:52.091457 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.091440    2586 scope.go:117] "RemoveContainer" containerID="ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888"
Sep 17 21:29:52.098329 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.098312    2586 scope.go:117] "RemoveContainer" containerID="a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28"
Sep 17 21:29:52.098616 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:52.098598    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28\": container with ID starting with a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28 not found: ID does not exist" containerID="a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28"
Sep 17 21:29:52.098695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.098623    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28"} err="failed to get container status \"a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28\": rpc error: code = NotFound desc = could not find container \"a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28\": container with ID starting with a0ba1ae093415373f0b9f1054a9029cd6db4d494ebf4bf190ffb275bb8d65b28 not found: ID does not exist"
Sep 17 21:29:52.098695 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.098639    2586 scope.go:117] "RemoveContainer" containerID="7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186"
Sep 17 21:29:52.098931 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:52.098910    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186\": container with ID starting with 7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186 not found: ID does not exist" containerID="7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186"
Sep 17 21:29:52.099012 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.098940    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186"} err="failed to get container status \"7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186\": rpc error: code = NotFound desc = could not find container \"7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186\": container with ID starting with 7b2b8b77c8272b5081d9c89872c23788c34e7e09aca74ae3ea9dbe36c223c186 not found: ID does not exist"
Sep 17 21:29:52.099012 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.098960    2586 scope.go:117] "RemoveContainer" containerID="ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888"
Sep 17 21:29:52.099291 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:29:52.099274    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888\": container with ID starting with ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888 not found: ID does not exist" containerID="ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888"
Sep 17 21:29:52.099333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.099297    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888"} err="failed to get container status \"ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888\": rpc error: code = NotFound desc = could not find container \"ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888\": container with ID starting with ddecd1df3caaef77a725b26de5e89cf53030117ae339ecb22e7cf13d11026888 not found: ID does not exist"
Sep 17 21:29:52.109575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.109547    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"]
Sep 17 21:29:52.110700 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.110682    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-j6wgd"]
Sep 17 21:29:52.122533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.122513    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:52.122533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.122535    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2x7t\" (UniqueName: \"kubernetes.io/projected/348b8f0f-d706-43f5-b022-16c572c5c06e-kube-api-access-r2x7t\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:52.122659 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.122544    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/348b8f0f-d706-43f5-b022-16c572c5c06e-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:52.122659 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.122553    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/348b8f0f-d706-43f5-b022-16c572c5c06e-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:52.122659 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.122560    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/348b8f0f-d706-43f5-b022-16c572c5c06e-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:29:52.670979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:52.670949    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" path="/var/lib/kubelet/pods/348b8f0f-d706-43f5-b022-16c572c5c06e/volumes"
Sep 17 21:29:53.079986 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:53.079903    2586 generic.go:356] "Generic (PLEG): container finished" podID="453681f5-8078-41b3-a4a1-12769d88ef54" containerID="a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b" exitCode=0
Sep 17 21:29:53.079986 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:53.079956    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerDied","Data":"a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b"}
Sep 17 21:29:54.085807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:54.085771    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerStarted","Data":"9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862"}
Sep 17 21:29:54.086170 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:54.085866    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerStarted","Data":"c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b"}
Sep 17 21:29:54.086170 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:54.086064    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:54.118210 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:54.118156    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podStartSLOduration=3.47020512 podStartE2EDuration="7.11814034s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.64793522s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:29:54.116867122 +0000 UTC m=+2409.650806931" watchObservedRunningTime="2026-09-17 21:29:54.11814034 +0000 UTC m=+2409.652080151"
Sep 17 21:29:54.996185 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:54.996148    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:29:55.090369 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:55.090326    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:29:55.091766 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:55.091729    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:29:56.093433 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:29:56.093391    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:30:01.103035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:01.102963    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:30:01.103417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:01.103387    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:30:04.996517 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:04.996481    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:30:11.103741 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:11.103702    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:30:14.996246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:14.996200    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:30:21.103862 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:21.103821    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:30:24.996151 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:24.996113    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:30:31.103406 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:31.103337    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:30:34.996279 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:34.996236    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:30:41.104382 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:41.104332    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:30:44.996205 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:44.996168    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 21:30:51.104075 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:51.104038    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:30:52.485733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:52.485707    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:30:52.487028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:52.487004    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:30:54.997069 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:54.997039    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:30:56.635804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:56.635775    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"]
Sep 17 21:30:56.636234 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:56.635974    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container" containerID="cri-o://fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181" gracePeriod=30
Sep 17 21:30:56.636234 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:56.636076    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kube-rbac-proxy" containerID="cri-o://93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7" gracePeriod=30
Sep 17 21:30:57.313870 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:57.313835    2586 generic.go:356] "Generic (PLEG): container finished" podID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerID="93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7" exitCode=2
Sep 17 21:30:57.314039 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:57.313911    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerDied","Data":"93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7"}
Sep 17 21:30:59.991219 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:30:59.991184    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 17 21:31:01.104436 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.104406    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:31:01.846140 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846066    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"]
Sep 17 21:31:01.846333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846322    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="storage-initializer"
Sep 17 21:31:01.846399 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846334    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="storage-initializer"
Sep 17 21:31:01.846399 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846345    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kserve-container"
Sep 17 21:31:01.846399 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846364    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kserve-container"
Sep 17 21:31:01.846399 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846377    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kube-rbac-proxy"
Sep 17 21:31:01.846399 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846385    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kube-rbac-proxy"
Sep 17 21:31:01.846560 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846434    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kserve-container"
Sep 17 21:31:01.846560 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.846444    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="348b8f0f-d706-43f5-b022-16c572c5c06e" containerName="kube-rbac-proxy"
Sep 17 21:31:01.849318 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.849303    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:01.851742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.851722    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 21:31:01.851909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.851888    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:31:01.859844 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.859825    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"]
Sep 17 21:31:01.912220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.912192    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c66c93-639d-4a82-a506-1208bc26dd2e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:01.912339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.912241    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3c66c93-639d-4a82-a506-1208bc26dd2e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:01.912339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.912305    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:01.912474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.912378    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqw9\" (UniqueName: \"kubernetes.io/projected/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-api-access-bmqw9\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:01.912474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:01.912437    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013023    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c66c93-639d-4a82-a506-1208bc26dd2e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013176 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3c66c93-639d-4a82-a506-1208bc26dd2e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013176 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013176 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013150    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqw9\" (UniqueName: \"kubernetes.io/projected/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-api-access-bmqw9\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013337 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013222    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013637    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3c66c93-639d-4a82-a506-1208bc26dd2e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013829 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013807    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.013897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.013859    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.015561 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.015541    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c66c93-639d-4a82-a506-1208bc26dd2e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.024731 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.024706    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqw9\" (UniqueName: \"kubernetes.io/projected/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-api-access-bmqw9\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-q247r\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.159869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.159844    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:02.176806 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.176785    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:31:02.315654 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.315621    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-proxy-tls\") pod \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") "
Sep 17 21:31:02.315835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.315672    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-rbac-proxy-sar-config\") pod \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") "
Sep 17 21:31:02.315835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.315694    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mww5x\" (UniqueName: \"kubernetes.io/projected/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-api-access-mww5x\") pod \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") "
Sep 17 21:31:02.315835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.315752    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-cabundle-cert\") pod \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") "
Sep 17 21:31:02.315835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.315769    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kserve-provision-location\") pod \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\" (UID: \"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7\") "
Sep 17 21:31:02.316135 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.316108    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" (UID: "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:31:02.316219 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.316138    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" (UID: "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:31:02.316219 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.316134    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" (UID: "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:31:02.317752 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.317731    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" (UID: "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:31:02.317829 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.317782    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-api-access-mww5x" (OuterVolumeSpecName: "kube-api-access-mww5x") pod "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" (UID: "b8a96751-7842-4ddd-9cc3-a59d2ebae7c7"). InnerVolumeSpecName "kube-api-access-mww5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:31:02.334220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.334192    2586 generic.go:356] "Generic (PLEG): container finished" podID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerID="fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181" exitCode=0
Sep 17 21:31:02.334333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.334240    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerDied","Data":"fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181"}
Sep 17 21:31:02.334333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.334250    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"
Sep 17 21:31:02.334333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.334293    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4" event={"ID":"b8a96751-7842-4ddd-9cc3-a59d2ebae7c7","Type":"ContainerDied","Data":"ca2921a47fc2a372b3f57bb37f2e499204e3ce1688b8e826edaa60f2b5926660"}
Sep 17 21:31:02.334333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.334322    2586 scope.go:117] "RemoveContainer" containerID="93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7"
Sep 17 21:31:02.342215 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.342199    2586 scope.go:117] "RemoveContainer" containerID="fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181"
Sep 17 21:31:02.349139 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.349124    2586 scope.go:117] "RemoveContainer" containerID="e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6"
Sep 17 21:31:02.355628 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.355607    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"]
Sep 17 21:31:02.356143 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.356131    2586 scope.go:117] "RemoveContainer" containerID="93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7"
Sep 17 21:31:02.356434 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:31:02.356413    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7\": container with ID starting with 93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7 not found: ID does not exist" containerID="93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7"
Sep 17 21:31:02.356525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.356445    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7"} err="failed to get container status \"93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7\": rpc error: code = NotFound desc = could not find container \"93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7\": container with ID starting with 93c6bbb822f354a8c430cc5b551f47979424922928c4198a80f93514f15a89a7 not found: ID does not exist"
Sep 17 21:31:02.356525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.356470    2586 scope.go:117] "RemoveContainer" containerID="fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181"
Sep 17 21:31:02.356732 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:31:02.356716    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181\": container with ID starting with fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181 not found: ID does not exist" containerID="fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181"
Sep 17 21:31:02.356776 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.356737    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181"} err="failed to get container status \"fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181\": rpc error: code = NotFound desc = could not find container \"fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181\": container with ID starting with fe9b6abe3874b2b0a2a41a9bd11a448c34a3f6d60943c254b508d7cc3ec85181 not found: ID does not exist"
Sep 17 21:31:02.356776 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.356750    2586 scope.go:117] "RemoveContainer" containerID="e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6"
Sep 17 21:31:02.361916 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:31:02.356948    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6\": container with ID starting with e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6 not found: ID does not exist" containerID="e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6"
Sep 17 21:31:02.361916 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.357396    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6"} err="failed to get container status \"e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6\": rpc error: code = NotFound desc = could not find container \"e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6\": container with ID starting with e4139f862ef89428c6e2531133883eafaf8fd48ffb53b926aef3fa0e022071c6 not found: ID does not exist"
Sep 17 21:31:02.364030 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.364005    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-7gfn4"]
Sep 17 21:31:02.416717 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.416665    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:02.416717 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.416689    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:02.416717 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.416704    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mww5x\" (UniqueName: \"kubernetes.io/projected/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kube-api-access-mww5x\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:02.416868 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.416718    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:02.416868 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.416731    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:02.431819 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.431797    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"]
Sep 17 21:31:02.434492 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:31:02.434471    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c66c93_639d_4a82_a506_1208bc26dd2e.slice/crio-0778fd5126f2006b0edfdcde0f18be6cadafca4204352d108e452cf011f1943a WatchSource:0}: Error finding container 0778fd5126f2006b0edfdcde0f18be6cadafca4204352d108e452cf011f1943a: Status 404 returned error can't find the container with id 0778fd5126f2006b0edfdcde0f18be6cadafca4204352d108e452cf011f1943a
Sep 17 21:31:02.670108 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:02.670034    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" path="/var/lib/kubelet/pods/b8a96751-7842-4ddd-9cc3-a59d2ebae7c7/volumes"
Sep 17 21:31:03.339451 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:03.339417    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerStarted","Data":"f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c"}
Sep 17 21:31:03.339910 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:03.339484    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerStarted","Data":"0778fd5126f2006b0edfdcde0f18be6cadafca4204352d108e452cf011f1943a"}
Sep 17 21:31:07.355082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:07.355053    2586 generic.go:356] "Generic (PLEG): container finished" podID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerID="f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c" exitCode=0
Sep 17 21:31:07.355475 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:07.355102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerDied","Data":"f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c"}
Sep 17 21:31:07.841058 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:07.841023    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"]
Sep 17 21:31:07.841264 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:07.841241    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" containerID="cri-o://c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b" gracePeriod=30
Sep 17 21:31:07.841346 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:07.841275    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kube-rbac-proxy" containerID="cri-o://9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862" gracePeriod=30
Sep 17 21:31:08.361207 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.361168    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerStarted","Data":"ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb"}
Sep 17 21:31:08.361674 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.361265    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerStarted","Data":"66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84"}
Sep 17 21:31:08.361674 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.361615    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:08.361674 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.361672    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:08.363012 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.362984    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:31:08.363454 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.363433    2586 generic.go:356] "Generic (PLEG): container finished" podID="453681f5-8078-41b3-a4a1-12769d88ef54" containerID="9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862" exitCode=2
Sep 17 21:31:08.363535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.363468    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerDied","Data":"9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862"}
Sep 17 21:31:08.381080 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:08.381038    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podStartSLOduration=3.473055376 podStartE2EDuration="7.381026456s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.90797108s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:31:08.380298291 +0000 UTC m=+2483.914238102" watchObservedRunningTime="2026-09-17 21:31:08.381026456 +0000 UTC m=+2483.914966265"
Sep 17 21:31:09.367665 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:09.367620    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:31:11.094511 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.094468    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 17 21:31:11.104151 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.104120    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 21:31:11.577484 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.577460    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:31:11.675858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.675797    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453681f5-8078-41b3-a4a1-12769d88ef54-kserve-provision-location\") pod \"453681f5-8078-41b3-a4a1-12769d88ef54\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") "
Sep 17 21:31:11.675858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.675824    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-kube-rbac-proxy-sar-config\") pod \"453681f5-8078-41b3-a4a1-12769d88ef54\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") "
Sep 17 21:31:11.675858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.675853    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls\") pod \"453681f5-8078-41b3-a4a1-12769d88ef54\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") "
Sep 17 21:31:11.676062 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.675888    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl6k6\" (UniqueName: \"kubernetes.io/projected/453681f5-8078-41b3-a4a1-12769d88ef54-kube-api-access-dl6k6\") pod \"453681f5-8078-41b3-a4a1-12769d88ef54\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") "
Sep 17 21:31:11.676062 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.676000    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-cabundle-cert\") pod \"453681f5-8078-41b3-a4a1-12769d88ef54\" (UID: \"453681f5-8078-41b3-a4a1-12769d88ef54\") "
Sep 17 21:31:11.676179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.676151    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453681f5-8078-41b3-a4a1-12769d88ef54-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "453681f5-8078-41b3-a4a1-12769d88ef54" (UID: "453681f5-8078-41b3-a4a1-12769d88ef54"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:31:11.676238 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.676181    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "453681f5-8078-41b3-a4a1-12769d88ef54" (UID: "453681f5-8078-41b3-a4a1-12769d88ef54"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:31:11.676415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.676390    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "453681f5-8078-41b3-a4a1-12769d88ef54" (UID: "453681f5-8078-41b3-a4a1-12769d88ef54"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:31:11.677852 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.677831    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "453681f5-8078-41b3-a4a1-12769d88ef54" (UID: "453681f5-8078-41b3-a4a1-12769d88ef54"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:31:11.677925 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.677889    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453681f5-8078-41b3-a4a1-12769d88ef54-kube-api-access-dl6k6" (OuterVolumeSpecName: "kube-api-access-dl6k6") pod "453681f5-8078-41b3-a4a1-12769d88ef54" (UID: "453681f5-8078-41b3-a4a1-12769d88ef54"). InnerVolumeSpecName "kube-api-access-dl6k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:31:11.777098 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.777058    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:11.777098 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.777094    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453681f5-8078-41b3-a4a1-12769d88ef54-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:11.777098 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.777109    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453681f5-8078-41b3-a4a1-12769d88ef54-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:11.777393 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.777121    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453681f5-8078-41b3-a4a1-12769d88ef54-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:11.777393 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:11.777130    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl6k6\" (UniqueName: \"kubernetes.io/projected/453681f5-8078-41b3-a4a1-12769d88ef54-kube-api-access-dl6k6\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:31:12.379967 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.379931    2586 generic.go:356] "Generic (PLEG): container finished" podID="453681f5-8078-41b3-a4a1-12769d88ef54" containerID="c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b" exitCode=0
Sep 17 21:31:12.380351 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.379999    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"
Sep 17 21:31:12.380351 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.380015    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerDied","Data":"c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b"}
Sep 17 21:31:12.380351 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.380079    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l" event={"ID":"453681f5-8078-41b3-a4a1-12769d88ef54","Type":"ContainerDied","Data":"622c22503162d6c056a6d25099797000c67a9ec546fc65459001cdee4cc1964e"}
Sep 17 21:31:12.380351 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.380106    2586 scope.go:117] "RemoveContainer" containerID="9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862"
Sep 17 21:31:12.388745 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.388729    2586 scope.go:117] "RemoveContainer" containerID="c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b"
Sep 17 21:31:12.395507 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.395492    2586 scope.go:117] "RemoveContainer" containerID="a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b"
Sep 17 21:31:12.401965 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.401949    2586 scope.go:117] "RemoveContainer" containerID="9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862"
Sep 17 21:31:12.402206 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:31:12.402190    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862\": container with ID starting with 9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862 not found: ID does not exist" containerID="9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862"
Sep 17 21:31:12.402247 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.402214    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862"} err="failed to get container status \"9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862\": rpc error: code = NotFound desc = could not find container \"9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862\": container with ID starting with 9e37e97cdea7d19b3734f45c43672848522e4bab7a0574815f16b2f136ee6862 not found: ID does not exist"
Sep 17 21:31:12.402247 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.402231    2586 scope.go:117] "RemoveContainer" containerID="c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b"
Sep 17 21:31:12.402479 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:31:12.402462    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b\": container with ID starting with c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b not found: ID does not exist" containerID="c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b"
Sep 17 21:31:12.402515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.402485    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b"} err="failed to get container status \"c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b\": rpc error: code = NotFound desc = could not find container \"c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b\": container with ID starting with c2f23fb99c7d2792ec57ea1de11668f2a81b664d929d6339f5747d86fa7f615b not found: ID does not exist"
Sep 17 21:31:12.402515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.402498    2586 scope.go:117] "RemoveContainer" containerID="a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b"
Sep 17 21:31:12.402712 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:31:12.402698    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b\": container with ID starting with a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b not found: ID does not exist" containerID="a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b"
Sep 17 21:31:12.402754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.402717    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b"} err="failed to get container status \"a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b\": rpc error: code = NotFound desc = could not find container \"a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b\": container with ID starting with a200fbc428a083883d3d302e5b8ae1819ee34d84bf0ec4ec26ee0f441c3b918b not found: ID does not exist"
Sep 17 21:31:12.408675 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.408654    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"]
Sep 17 21:31:12.415305 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.415283    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-pcj2l"]
Sep 17 21:31:12.670205 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:12.670132    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" path="/var/lib/kubelet/pods/453681f5-8078-41b3-a4a1-12769d88ef54/volumes"
Sep 17 21:31:13.032471 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032389    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"]
Sep 17 21:31:13.032669 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032657    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kube-rbac-proxy"
Sep 17 21:31:13.032714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032670    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kube-rbac-proxy"
Sep 17 21:31:13.032714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032680    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="storage-initializer"
Sep 17 21:31:13.032714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032685    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="storage-initializer"
Sep 17 21:31:13.032714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032694    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kube-rbac-proxy"
Sep 17 21:31:13.032714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032699    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kube-rbac-proxy"
Sep 17 21:31:13.032714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032708    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032716    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032728    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="storage-initializer"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032733    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="storage-initializer"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032739    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032743    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032790    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kube-rbac-proxy"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032802    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a96751-7842-4ddd-9cc3-a59d2ebae7c7" containerName="kserve-container"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032813    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kserve-container"
Sep 17 21:31:13.032896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.032818    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="453681f5-8078-41b3-a4a1-12769d88ef54" containerName="kube-rbac-proxy"
Sep 17 21:31:13.037317 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.037297    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.039875 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.039857    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 21:31:13.039967 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.039898    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:31:13.048919 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.048891    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"]
Sep 17 21:31:13.186770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.186742    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbk6\" (UniqueName: \"kubernetes.io/projected/170787e6-359f-4103-b0b5-113302af5efb-kube-api-access-bdbk6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.186926 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.186781    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.186926 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.186879    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.186926 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.186920    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170787e6-359f-4103-b0b5-113302af5efb-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.187062 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.186947    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170787e6-359f-4103-b0b5-113302af5efb-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288095 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288095 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288051    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170787e6-359f-4103-b0b5-113302af5efb-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170787e6-359f-4103-b0b5-113302af5efb-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288152    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbk6\" (UniqueName: \"kubernetes.io/projected/170787e6-359f-4103-b0b5-113302af5efb-kube-api-access-bdbk6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288171    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288564    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170787e6-359f-4103-b0b5-113302af5efb-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288834 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288814    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.288944 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.288922    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.290596 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.290579    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170787e6-359f-4103-b0b5-113302af5efb-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.296746 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.296726    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbk6\" (UniqueName: \"kubernetes.io/projected/170787e6-359f-4103-b0b5-113302af5efb-kube-api-access-bdbk6\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.346893 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.346872    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:13.617006 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:13.616969    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"]
Sep 17 21:31:13.620552 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:31:13.620522    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170787e6_359f_4103_b0b5_113302af5efb.slice/crio-df8791f16e60459764d580d6c3e4d75e52cb0b22675f1c0bac27d0e60d4532c3 WatchSource:0}: Error finding container df8791f16e60459764d580d6c3e4d75e52cb0b22675f1c0bac27d0e60d4532c3: Status 404 returned error can't find the container with id df8791f16e60459764d580d6c3e4d75e52cb0b22675f1c0bac27d0e60d4532c3
Sep 17 21:31:14.372213 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:14.372187    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:31:14.372715 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:14.372692    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:31:14.391483 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:14.391456    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerStarted","Data":"0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839"}
Sep 17 21:31:14.391630 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:14.391519    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerStarted","Data":"df8791f16e60459764d580d6c3e4d75e52cb0b22675f1c0bac27d0e60d4532c3"}
Sep 17 21:31:18.408284 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:18.408245    2586 generic.go:356] "Generic (PLEG): container finished" podID="170787e6-359f-4103-b0b5-113302af5efb" containerID="0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839" exitCode=0
Sep 17 21:31:18.408831 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:18.408303    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerDied","Data":"0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839"}
Sep 17 21:31:19.414475 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:19.414443    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerStarted","Data":"3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7"}
Sep 17 21:31:19.414957 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:19.414505    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerStarted","Data":"5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc"}
Sep 17 21:31:19.414957 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:19.414694    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:19.435277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:19.435235    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" podStartSLOduration=2.469202086 podStartE2EDuration="6.435220588s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.966018502s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:31:19.434189609 +0000 UTC m=+2494.968129434" watchObservedRunningTime="2026-09-17 21:31:19.435220588 +0000 UTC m=+2494.969160397"
Sep 17 21:31:20.419871 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:20.419840    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:24.373170 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:24.373128    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:31:26.428955 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:26.428926    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:31:34.373495 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:34.373453    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:31:44.373482 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:44.373440    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:31:54.373648 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:54.373604    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:31:56.511752 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:31:56.511707    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 21:32:04.373444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:04.373405    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:32:06.431556 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:06.431527    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:32:13.112836 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:13.112801    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"]
Sep 17 21:32:13.113196 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:13.113086    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kserve-container" containerID="cri-o://5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc" gracePeriod=30
Sep 17 21:32:13.113196 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:13.113127    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kube-rbac-proxy" containerID="cri-o://3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7" gracePeriod=30
Sep 17 21:32:13.619912 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:13.619881    2586 generic.go:356] "Generic (PLEG): container finished" podID="170787e6-359f-4103-b0b5-113302af5efb" containerID="3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7" exitCode=2
Sep 17 21:32:13.620085 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:13.619955    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerDied","Data":"3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7"}
Sep 17 21:32:14.372967 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:14.372931    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:32:16.424670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:16.424635    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 21:32:18.329716 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.329682    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"]
Sep 17 21:32:18.332926 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.332910    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.336785 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.336765    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 21:32:18.336908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.336772    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 21:32:18.349786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.349766    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"]
Sep 17 21:32:18.448452 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.448420    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5746400-f112-4ca9-8bae-43494ee73af4-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.448610 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.448463    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpg5\" (UniqueName: \"kubernetes.io/projected/b5746400-f112-4ca9-8bae-43494ee73af4-kube-api-access-kbpg5\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.448610 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.448519    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.448610 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.448569    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.448748 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.448613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549096    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5746400-f112-4ca9-8bae-43494ee73af4-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549146    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpg5\" (UniqueName: \"kubernetes.io/projected/b5746400-f112-4ca9-8bae-43494ee73af4-kube-api-access-kbpg5\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549265    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549323    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549383    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549607 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:18.549532    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 21:32:18.549607 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:18.549605    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls podName:b5746400-f112-4ca9-8bae-43494ee73af4 nodeName:}" failed. No retries permitted until 2026-09-17 21:32:19.049581552 +0000 UTC m=+2554.583521354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-qk2b5" (UID: "b5746400-f112-4ca9-8bae-43494ee73af4") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 21:32:18.549718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549598    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5746400-f112-4ca9-8bae-43494ee73af4-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549912    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.549979 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.549950    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:18.560270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:18.560248    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpg5\" (UniqueName: \"kubernetes.io/projected/b5746400-f112-4ca9-8bae-43494ee73af4-kube-api-access-kbpg5\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:19.053150 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:19.053109    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:19.055661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:19.055627    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-qk2b5\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:19.242401 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:19.242343    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:19.521120 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:19.520981    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"]
Sep 17 21:32:19.523765 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:32:19.523736    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5746400_f112_4ca9_8bae_43494ee73af4.slice/crio-6f5f8c81294c7a77dcce82cb26260b9f03189a53eb4f0a6b0f31d6b41807d3e2 WatchSource:0}: Error finding container 6f5f8c81294c7a77dcce82cb26260b9f03189a53eb4f0a6b0f31d6b41807d3e2: Status 404 returned error can't find the container with id 6f5f8c81294c7a77dcce82cb26260b9f03189a53eb4f0a6b0f31d6b41807d3e2
Sep 17 21:32:19.641769 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:19.641730    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerStarted","Data":"01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872"}
Sep 17 21:32:19.641920 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:19.641800    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerStarted","Data":"6f5f8c81294c7a77dcce82cb26260b9f03189a53eb4f0a6b0f31d6b41807d3e2"}
Sep 17 21:32:20.164423 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.164392    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:32:20.262919 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.262876    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-cabundle-cert\") pod \"170787e6-359f-4103-b0b5-113302af5efb\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") "
Sep 17 21:32:20.263082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.262927    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-kube-rbac-proxy-sar-config\") pod \"170787e6-359f-4103-b0b5-113302af5efb\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") "
Sep 17 21:32:20.263082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.262958    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdbk6\" (UniqueName: \"kubernetes.io/projected/170787e6-359f-4103-b0b5-113302af5efb-kube-api-access-bdbk6\") pod \"170787e6-359f-4103-b0b5-113302af5efb\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") "
Sep 17 21:32:20.263082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.262993    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170787e6-359f-4103-b0b5-113302af5efb-proxy-tls\") pod \"170787e6-359f-4103-b0b5-113302af5efb\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") "
Sep 17 21:32:20.263082 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.263040    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170787e6-359f-4103-b0b5-113302af5efb-kserve-provision-location\") pod \"170787e6-359f-4103-b0b5-113302af5efb\" (UID: \"170787e6-359f-4103-b0b5-113302af5efb\") "
Sep 17 21:32:20.263402 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.263351    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "170787e6-359f-4103-b0b5-113302af5efb" (UID: "170787e6-359f-4103-b0b5-113302af5efb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:32:20.263467 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.263445    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "170787e6-359f-4103-b0b5-113302af5efb" (UID: "170787e6-359f-4103-b0b5-113302af5efb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:32:20.263555 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.263533    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170787e6-359f-4103-b0b5-113302af5efb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "170787e6-359f-4103-b0b5-113302af5efb" (UID: "170787e6-359f-4103-b0b5-113302af5efb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:32:20.265065 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.265007    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170787e6-359f-4103-b0b5-113302af5efb-kube-api-access-bdbk6" (OuterVolumeSpecName: "kube-api-access-bdbk6") pod "170787e6-359f-4103-b0b5-113302af5efb" (UID: "170787e6-359f-4103-b0b5-113302af5efb"). InnerVolumeSpecName "kube-api-access-bdbk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:32:20.265065 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.265017    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170787e6-359f-4103-b0b5-113302af5efb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "170787e6-359f-4103-b0b5-113302af5efb" (UID: "170787e6-359f-4103-b0b5-113302af5efb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:32:20.364516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.364491    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170787e6-359f-4103-b0b5-113302af5efb-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:20.364516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.364511    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:20.364671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.364520    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170787e6-359f-4103-b0b5-113302af5efb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:20.364671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.364531    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdbk6\" (UniqueName: \"kubernetes.io/projected/170787e6-359f-4103-b0b5-113302af5efb-kube-api-access-bdbk6\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:20.364671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.364540    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170787e6-359f-4103-b0b5-113302af5efb-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:20.647325 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.647293    2586 generic.go:356] "Generic (PLEG): container finished" podID="170787e6-359f-4103-b0b5-113302af5efb" containerID="5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc" exitCode=0
Sep 17 21:32:20.647728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.647351    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"
Sep 17 21:32:20.647728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.647386    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerDied","Data":"5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc"}
Sep 17 21:32:20.647728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.647454    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj" event={"ID":"170787e6-359f-4103-b0b5-113302af5efb","Type":"ContainerDied","Data":"df8791f16e60459764d580d6c3e4d75e52cb0b22675f1c0bac27d0e60d4532c3"}
Sep 17 21:32:20.647728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.647493    2586 scope.go:117] "RemoveContainer" containerID="3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7"
Sep 17 21:32:20.657538 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.657301    2586 scope.go:117] "RemoveContainer" containerID="5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc"
Sep 17 21:32:20.664619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.664603    2586 scope.go:117] "RemoveContainer" containerID="0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839"
Sep 17 21:32:20.671786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.671774    2586 scope.go:117] "RemoveContainer" containerID="3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7"
Sep 17 21:32:20.672032 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:20.672012    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7\": container with ID starting with 3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7 not found: ID does not exist" containerID="3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7"
Sep 17 21:32:20.672074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.672042    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7"} err="failed to get container status \"3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7\": rpc error: code = NotFound desc = could not find container \"3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7\": container with ID starting with 3bb164d1a6638feeb09e5ef1384c7a902308c52f00589f7f34fdaf7c25e89ba7 not found: ID does not exist"
Sep 17 21:32:20.672074 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.672059    2586 scope.go:117] "RemoveContainer" containerID="5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc"
Sep 17 21:32:20.672278 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:20.672261    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc\": container with ID starting with 5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc not found: ID does not exist" containerID="5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc"
Sep 17 21:32:20.672321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.672284    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc"} err="failed to get container status \"5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc\": rpc error: code = NotFound desc = could not find container \"5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc\": container with ID starting with 5e552475871a1f394acda768a93314c95b87458c9cbf981f7e8eef65657f31dc not found: ID does not exist"
Sep 17 21:32:20.672321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.672297    2586 scope.go:117] "RemoveContainer" containerID="0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839"
Sep 17 21:32:20.672567 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:20.672552    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839\": container with ID starting with 0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839 not found: ID does not exist" containerID="0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839"
Sep 17 21:32:20.672619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.672573    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839"} err="failed to get container status \"0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839\": rpc error: code = NotFound desc = could not find container \"0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839\": container with ID starting with 0f17c9917644c602d3cb1e0fb28c87b6e7a6164156bb57f149897d277ba1a839 not found: ID does not exist"
Sep 17 21:32:20.684783 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.684761    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"]
Sep 17 21:32:20.689156 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:20.689138    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tvxzj"]
Sep 17 21:32:22.674781 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:22.674746    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="170787e6-359f-4103-b0b5-113302af5efb" path="/var/lib/kubelet/pods/170787e6-359f-4103-b0b5-113302af5efb/volumes"
Sep 17 21:32:23.660627 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:23.660595    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5746400-f112-4ca9-8bae-43494ee73af4" containerID="01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872" exitCode=0
Sep 17 21:32:23.660832 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:23.660643    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerDied","Data":"01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872"}
Sep 17 21:32:24.372645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:24.372610    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:32:24.669960 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:24.669874    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerStarted","Data":"a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd"}
Sep 17 21:32:24.669960 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:24.669942    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerStarted","Data":"819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b"}
Sep 17 21:32:24.670205 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:24.670181    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:24.670270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:24.670234    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:24.671624 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:24.671601    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:32:24.703384 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:24.703325    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podStartSLOduration=2.93685534 podStartE2EDuration="6.703312203s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.766456863s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:32:24.702742275 +0000 UTC m=+2560.236682086" watchObservedRunningTime="2026-09-17 21:32:24.703312203 +0000 UTC m=+2560.237252011"
Sep 17 21:32:25.670209 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:25.670171    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:32:30.674672 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:30.674643    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:32:30.675111 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:30.675085    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:32:34.372731 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:34.372696    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:32:35.667056 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:35.667027    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:32:40.675567 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:40.675532    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:32:43.237222 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:43.237187    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"]
Sep 17 21:32:43.237645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:43.237423    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" containerID="cri-o://66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84" gracePeriod=30
Sep 17 21:32:43.237645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:43.237441    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kube-rbac-proxy" containerID="cri-o://ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb" gracePeriod=30
Sep 17 21:32:43.737765 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:43.737732    2586 generic.go:356] "Generic (PLEG): container finished" podID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerID="ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb" exitCode=2
Sep 17 21:32:43.737927 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:43.737799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerDied","Data":"ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb"}
Sep 17 21:32:44.368688 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:44.368648    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 17 21:32:45.666854 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:45.666808    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 21:32:48.392503 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.392470    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"]
Sep 17 21:32:48.392931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.392874    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="storage-initializer"
Sep 17 21:32:48.392931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.392893    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="storage-initializer"
Sep 17 21:32:48.392931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.392915    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kserve-container"
Sep 17 21:32:48.392931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.392923    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kserve-container"
Sep 17 21:32:48.393161 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.392943    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kube-rbac-proxy"
Sep 17 21:32:48.393161 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.392951    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kube-rbac-proxy"
Sep 17 21:32:48.393161 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.393026    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kube-rbac-proxy"
Sep 17 21:32:48.393161 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.393042    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="170787e6-359f-4103-b0b5-113302af5efb" containerName="kserve-container"
Sep 17 21:32:48.396091 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.396068    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.398487 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.398468    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 21:32:48.398590 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.398567    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:32:48.409402 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.409381    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"]
Sep 17 21:32:48.472532 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.472511    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:32:48.566001 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.565973    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmqw9\" (UniqueName: \"kubernetes.io/projected/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-api-access-bmqw9\") pod \"a3c66c93-639d-4a82-a506-1208bc26dd2e\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") "
Sep 17 21:32:48.566146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566014    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3c66c93-639d-4a82-a506-1208bc26dd2e-kserve-provision-location\") pod \"a3c66c93-639d-4a82-a506-1208bc26dd2e\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") "
Sep 17 21:32:48.566146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566051    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-cabundle-cert\") pod \"a3c66c93-639d-4a82-a506-1208bc26dd2e\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") "
Sep 17 21:32:48.566146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566075    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c66c93-639d-4a82-a506-1208bc26dd2e-proxy-tls\") pod \"a3c66c93-639d-4a82-a506-1208bc26dd2e\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") "
Sep 17 21:32:48.566146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566094    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-rbac-proxy-sar-config\") pod \"a3c66c93-639d-4a82-a506-1208bc26dd2e\" (UID: \"a3c66c93-639d-4a82-a506-1208bc26dd2e\") "
Sep 17 21:32:48.566326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566185    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c93d49-7ff8-49f4-9931-e12eadb52f96-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.566326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566233    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.566326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566321    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.566515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566401    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.566515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566431    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-api-access-rfxhq\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.566515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566438    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3c66c93-639d-4a82-a506-1208bc26dd2e" (UID: "a3c66c93-639d-4a82-a506-1208bc26dd2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:32:48.566515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566466    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3c66c93-639d-4a82-a506-1208bc26dd2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3c66c93-639d-4a82-a506-1208bc26dd2e" (UID: "a3c66c93-639d-4a82-a506-1208bc26dd2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:32:48.566667 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.566511    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3c66c93-639d-4a82-a506-1208bc26dd2e" (UID: "a3c66c93-639d-4a82-a506-1208bc26dd2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:32:48.568145 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.568124    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-api-access-bmqw9" (OuterVolumeSpecName: "kube-api-access-bmqw9") pod "a3c66c93-639d-4a82-a506-1208bc26dd2e" (UID: "a3c66c93-639d-4a82-a506-1208bc26dd2e"). InnerVolumeSpecName "kube-api-access-bmqw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:32:48.568201 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.568175    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c66c93-639d-4a82-a506-1208bc26dd2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3c66c93-639d-4a82-a506-1208bc26dd2e" (UID: "a3c66c93-639d-4a82-a506-1208bc26dd2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:32:48.667474 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667446    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-api-access-rfxhq\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.667617 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667480    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c93d49-7ff8-49f4-9931-e12eadb52f96-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.667617 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667504    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.667617 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667539    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.667817 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667695    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.667817 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667760    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:48.667817 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667779    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3c66c93-639d-4a82-a506-1208bc26dd2e-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:48.667817 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667793    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:48.667817 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667809    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqw9\" (UniqueName: \"kubernetes.io/projected/a3c66c93-639d-4a82-a506-1208bc26dd2e-kube-api-access-bmqw9\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:48.668090 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.667822    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3c66c93-639d-4a82-a506-1208bc26dd2e-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:32:48.668262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.668240    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.668403 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.668318    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.668535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.668511    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.670038 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.670020    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c93d49-7ff8-49f4-9931-e12eadb52f96-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.684673 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.684651    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-api-access-rfxhq\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.706595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.706575    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:48.761692 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.761657    2586 generic.go:356] "Generic (PLEG): container finished" podID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerID="66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84" exitCode=0
Sep 17 21:32:48.761836 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.761714    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerDied","Data":"66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84"}
Sep 17 21:32:48.761836 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.761781    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r" event={"ID":"a3c66c93-639d-4a82-a506-1208bc26dd2e","Type":"ContainerDied","Data":"0778fd5126f2006b0edfdcde0f18be6cadafca4204352d108e452cf011f1943a"}
Sep 17 21:32:48.761836 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.761820    2586 scope.go:117] "RemoveContainer" containerID="ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb"
Sep 17 21:32:48.762005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.761991    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"
Sep 17 21:32:48.772063 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.772039    2586 scope.go:117] "RemoveContainer" containerID="66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84"
Sep 17 21:32:48.779985 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.779900    2586 scope.go:117] "RemoveContainer" containerID="f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c"
Sep 17 21:32:48.782341 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.782312    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"]
Sep 17 21:32:48.783925 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.783901    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-q247r"]
Sep 17 21:32:48.787171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.787154    2586 scope.go:117] "RemoveContainer" containerID="ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb"
Sep 17 21:32:48.787438 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:48.787419    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb\": container with ID starting with ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb not found: ID does not exist" containerID="ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb"
Sep 17 21:32:48.787508 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.787451    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb"} err="failed to get container status \"ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb\": rpc error: code = NotFound desc = could not find container \"ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb\": container with ID starting with ce65ac9204a3a9c5cc5e230d547ed56379a019c58c49b2b223a6e401594fa2cb not found: ID does not exist"
Sep 17 21:32:48.787508 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.787475    2586 scope.go:117] "RemoveContainer" containerID="66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84"
Sep 17 21:32:48.787743 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:48.787715    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84\": container with ID starting with 66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84 not found: ID does not exist" containerID="66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84"
Sep 17 21:32:48.787788 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.787741    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84"} err="failed to get container status \"66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84\": rpc error: code = NotFound desc = could not find container \"66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84\": container with ID starting with 66f395e084caab0cc284dd464fc4acdad598545457ce1e9f0ecc7f944de5db84 not found: ID does not exist"
Sep 17 21:32:48.787788 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.787761    2586 scope.go:117] "RemoveContainer" containerID="f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c"
Sep 17 21:32:48.787983 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:32:48.787965    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c\": container with ID starting with f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c not found: ID does not exist" containerID="f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c"
Sep 17 21:32:48.788042 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.787991    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c"} err="failed to get container status \"f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c\": rpc error: code = NotFound desc = could not find container \"f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c\": container with ID starting with f22bbd4922f34e95898f4d33ed279e83c29c000fa5925cc5b72cdb831aa2d02c not found: ID does not exist"
Sep 17 21:32:48.988820 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:48.988737    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"]
Sep 17 21:32:48.991294 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:32:48.991263    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c93d49_7ff8_49f4_9931_e12eadb52f96.slice/crio-6bfd3f1ff57afcb80435255028e638398e652d7663d4d939b32c68c59be6659e WatchSource:0}: Error finding container 6bfd3f1ff57afcb80435255028e638398e652d7663d4d939b32c68c59be6659e: Status 404 returned error can't find the container with id 6bfd3f1ff57afcb80435255028e638398e652d7663d4d939b32c68c59be6659e
Sep 17 21:32:49.767260 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:49.767219    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerStarted","Data":"47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd"}
Sep 17 21:32:49.767753 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:49.767295    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerStarted","Data":"6bfd3f1ff57afcb80435255028e638398e652d7663d4d939b32c68c59be6659e"}
Sep 17 21:32:50.671140 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:50.671100    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" path="/var/lib/kubelet/pods/a3c66c93-639d-4a82-a506-1208bc26dd2e/volumes"
Sep 17 21:32:50.675641 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:50.675610    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:32:53.785809 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:53.785775    2586 generic.go:356] "Generic (PLEG): container finished" podID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerID="47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd" exitCode=0
Sep 17 21:32:53.786227 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:53.785845    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerDied","Data":"47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd"}
Sep 17 21:32:54.792106 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:54.792075    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerStarted","Data":"96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a"}
Sep 17 21:32:54.792518 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:54.792133    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerStarted","Data":"ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f"}
Sep 17 21:32:54.792518 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:54.792396    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:32:54.834660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:54.834613    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" podStartSLOduration=2.972562978 podStartE2EDuration="6.834597151s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.862034173s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:32:54.833694195 +0000 UTC m=+2590.367634015" watchObservedRunningTime="2026-09-17 21:32:54.834597151 +0000 UTC m=+2590.368536959"
Sep 17 21:32:55.796308 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:32:55.796279    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:33:00.675292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:00.675212    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:33:01.804687 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:01.804655    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:33:10.675973 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:10.675932    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:33:20.675824 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:20.675784    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:33:30.676083 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:30.676055    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:33:31.808597 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:31.808569    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:33:38.400243 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:38.400210    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"]
Sep 17 21:33:38.400679 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:38.400476    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kserve-container" containerID="cri-o://ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f" gracePeriod=30
Sep 17 21:33:38.400679 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:38.400505    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kube-rbac-proxy" containerID="cri-o://96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a" gracePeriod=30
Sep 17 21:33:38.953898 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:38.953866    2586 generic.go:356] "Generic (PLEG): container finished" podID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerID="96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a" exitCode=2
Sep 17 21:33:38.954066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:38.953940    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerDied","Data":"96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a"}
Sep 17 21:33:39.061213 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.061171    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"]
Sep 17 21:33:39.061494 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.061460    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" containerID="cri-o://819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b" gracePeriod=30
Sep 17 21:33:39.061662 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.061514    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kube-rbac-proxy" containerID="cri-o://a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd" gracePeriod=30
Sep 17 21:33:39.340842 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.340818    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:33:39.422050 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422023    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-cabundle-cert\") pod \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") "
Sep 17 21:33:39.422510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422084    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c93d49-7ff8-49f4-9931-e12eadb52f96-proxy-tls\") pod \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") "
Sep 17 21:33:39.422510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422103    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-rbac-proxy-sar-config\") pod \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") "
Sep 17 21:33:39.422510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422129    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kserve-provision-location\") pod \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") "
Sep 17 21:33:39.422510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422154    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-api-access-rfxhq\") pod \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\" (UID: \"e8c93d49-7ff8-49f4-9931-e12eadb52f96\") "
Sep 17 21:33:39.422510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422470    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e8c93d49-7ff8-49f4-9931-e12eadb52f96" (UID: "e8c93d49-7ff8-49f4-9931-e12eadb52f96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:39.422510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422470    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e8c93d49-7ff8-49f4-9931-e12eadb52f96" (UID: "e8c93d49-7ff8-49f4-9931-e12eadb52f96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:33:39.422749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.422510    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e8c93d49-7ff8-49f4-9931-e12eadb52f96" (UID: "e8c93d49-7ff8-49f4-9931-e12eadb52f96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:33:39.424234 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.424209    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-api-access-rfxhq" (OuterVolumeSpecName: "kube-api-access-rfxhq") pod "e8c93d49-7ff8-49f4-9931-e12eadb52f96" (UID: "e8c93d49-7ff8-49f4-9931-e12eadb52f96"). InnerVolumeSpecName "kube-api-access-rfxhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:33:39.424234 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.424212    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c93d49-7ff8-49f4-9931-e12eadb52f96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8c93d49-7ff8-49f4-9931-e12eadb52f96" (UID: "e8c93d49-7ff8-49f4-9931-e12eadb52f96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:33:39.523603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.523533    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8c93d49-7ff8-49f4-9931-e12eadb52f96-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:39.523603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.523556    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:39.523603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.523565    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:39.523603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.523575    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxhq\" (UniqueName: \"kubernetes.io/projected/e8c93d49-7ff8-49f4-9931-e12eadb52f96-kube-api-access-rfxhq\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:39.523603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.523583    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e8c93d49-7ff8-49f4-9931-e12eadb52f96-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:39.959081 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.959044    2586 generic.go:356] "Generic (PLEG): container finished" podID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerID="ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f" exitCode=0
Sep 17 21:33:39.959273 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.959112    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"
Sep 17 21:33:39.959273 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.959138    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerDied","Data":"ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f"}
Sep 17 21:33:39.959273 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.959212    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld" event={"ID":"e8c93d49-7ff8-49f4-9931-e12eadb52f96","Type":"ContainerDied","Data":"6bfd3f1ff57afcb80435255028e638398e652d7663d4d939b32c68c59be6659e"}
Sep 17 21:33:39.959273 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.959248    2586 scope.go:117] "RemoveContainer" containerID="96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a"
Sep 17 21:33:39.961656 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.961634    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5746400-f112-4ca9-8bae-43494ee73af4" containerID="a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd" exitCode=2
Sep 17 21:33:39.961783 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.961681    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerDied","Data":"a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd"}
Sep 17 21:33:39.967479 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.967462    2586 scope.go:117] "RemoveContainer" containerID="ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f"
Sep 17 21:33:39.974433 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.974416    2586 scope.go:117] "RemoveContainer" containerID="47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd"
Sep 17 21:33:39.981271 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.981164    2586 scope.go:117] "RemoveContainer" containerID="96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a"
Sep 17 21:33:39.981577 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:33:39.981547    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a\": container with ID starting with 96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a not found: ID does not exist" containerID="96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a"
Sep 17 21:33:39.981694 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.981582    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a"} err="failed to get container status \"96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a\": rpc error: code = NotFound desc = could not find container \"96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a\": container with ID starting with 96cfbccf3e5d9177368b053e17d2172d3d7cd4f9db592c138dd3c2f681d5a20a not found: ID does not exist"
Sep 17 21:33:39.981694 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.981605    2586 scope.go:117] "RemoveContainer" containerID="ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f"
Sep 17 21:33:39.981955 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:33:39.981938    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f\": container with ID starting with ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f not found: ID does not exist" containerID="ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f"
Sep 17 21:33:39.982006 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.981958    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f"} err="failed to get container status \"ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f\": rpc error: code = NotFound desc = could not find container \"ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f\": container with ID starting with ac55960fadea889a2a1fd612cf7046b8f88064e29a65021eb8927ce3ade38b7f not found: ID does not exist"
Sep 17 21:33:39.982006 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.981972    2586 scope.go:117] "RemoveContainer" containerID="47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd"
Sep 17 21:33:39.982221 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:33:39.982203    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd\": container with ID starting with 47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd not found: ID does not exist" containerID="47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd"
Sep 17 21:33:39.982268 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.982223    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd"} err="failed to get container status \"47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd\": rpc error: code = NotFound desc = could not find container \"47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd\": container with ID starting with 47313f126e978b9733d411ad54c09ea44e60531fa866608ec59ccff6d7b89bdd not found: ID does not exist"
Sep 17 21:33:39.982988 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.982970    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"]
Sep 17 21:33:39.987053 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:39.987029    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s84ld"]
Sep 17 21:33:40.670710 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:40.670669    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 17 21:33:40.671536 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:40.671515    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" path="/var/lib/kubelet/pods/e8c93d49-7ff8-49f4-9931-e12eadb52f96/volumes"
Sep 17 21:33:40.675052 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:40.675033    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 21:33:42.796893 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.796870    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:33:42.845954 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.845922    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-cabundle-cert\") pod \"b5746400-f112-4ca9-8bae-43494ee73af4\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") "
Sep 17 21:33:42.846132 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.845991    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbpg5\" (UniqueName: \"kubernetes.io/projected/b5746400-f112-4ca9-8bae-43494ee73af4-kube-api-access-kbpg5\") pod \"b5746400-f112-4ca9-8bae-43494ee73af4\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") "
Sep 17 21:33:42.846132 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.846031    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5746400-f112-4ca9-8bae-43494ee73af4-kserve-provision-location\") pod \"b5746400-f112-4ca9-8bae-43494ee73af4\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") "
Sep 17 21:33:42.846132 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.846054    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls\") pod \"b5746400-f112-4ca9-8bae-43494ee73af4\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") "
Sep 17 21:33:42.846132 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.846105    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-kube-rbac-proxy-sar-config\") pod \"b5746400-f112-4ca9-8bae-43494ee73af4\" (UID: \"b5746400-f112-4ca9-8bae-43494ee73af4\") "
Sep 17 21:33:42.846343 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.846314    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5746400-f112-4ca9-8bae-43494ee73af4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b5746400-f112-4ca9-8bae-43494ee73af4" (UID: "b5746400-f112-4ca9-8bae-43494ee73af4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:33:42.846442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.846317    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b5746400-f112-4ca9-8bae-43494ee73af4" (UID: "b5746400-f112-4ca9-8bae-43494ee73af4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:33:42.846553 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.846516    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b5746400-f112-4ca9-8bae-43494ee73af4" (UID: "b5746400-f112-4ca9-8bae-43494ee73af4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:33:42.848136 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.848115    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5746400-f112-4ca9-8bae-43494ee73af4" (UID: "b5746400-f112-4ca9-8bae-43494ee73af4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:33:42.848206 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.848118    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5746400-f112-4ca9-8bae-43494ee73af4-kube-api-access-kbpg5" (OuterVolumeSpecName: "kube-api-access-kbpg5") pod "b5746400-f112-4ca9-8bae-43494ee73af4" (UID: "b5746400-f112-4ca9-8bae-43494ee73af4"). InnerVolumeSpecName "kube-api-access-kbpg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:33:42.946594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.946538    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpg5\" (UniqueName: \"kubernetes.io/projected/b5746400-f112-4ca9-8bae-43494ee73af4-kube-api-access-kbpg5\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:42.946594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.946557    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b5746400-f112-4ca9-8bae-43494ee73af4-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:42.946594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.946567    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5746400-f112-4ca9-8bae-43494ee73af4-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:42.946594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.946576    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:42.946594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.946585    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b5746400-f112-4ca9-8bae-43494ee73af4-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:33:42.974975 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.974951    2586 generic.go:356] "Generic (PLEG): container finished" podID="b5746400-f112-4ca9-8bae-43494ee73af4" containerID="819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b" exitCode=0
Sep 17 21:33:42.975065 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.974995    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerDied","Data":"819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b"}
Sep 17 21:33:42.975065 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.975014    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"
Sep 17 21:33:42.975065 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.975048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5" event={"ID":"b5746400-f112-4ca9-8bae-43494ee73af4","Type":"ContainerDied","Data":"6f5f8c81294c7a77dcce82cb26260b9f03189a53eb4f0a6b0f31d6b41807d3e2"}
Sep 17 21:33:42.975180 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.975073    2586 scope.go:117] "RemoveContainer" containerID="a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd"
Sep 17 21:33:42.982962 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.982873    2586 scope.go:117] "RemoveContainer" containerID="819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b"
Sep 17 21:33:42.989925 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.989910    2586 scope.go:117] "RemoveContainer" containerID="01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872"
Sep 17 21:33:42.998289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:42.998264    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"]
Sep 17 21:33:43.000654 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.000638    2586 scope.go:117] "RemoveContainer" containerID="a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd"
Sep 17 21:33:43.000883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.000850    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-qk2b5"]
Sep 17 21:33:43.001061 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:33:43.001030    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd\": container with ID starting with a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd not found: ID does not exist" containerID="a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd"
Sep 17 21:33:43.001105 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.001080    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd"} err="failed to get container status \"a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd\": rpc error: code = NotFound desc = could not find container \"a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd\": container with ID starting with a408eb03d44ae3aecfd5557f2c7ba3c7869fb1ba7edfa746da698ddd8b0967dd not found: ID does not exist"
Sep 17 21:33:43.001146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.001112    2586 scope.go:117] "RemoveContainer" containerID="819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b"
Sep 17 21:33:43.001458 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:33:43.001431    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b\": container with ID starting with 819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b not found: ID does not exist" containerID="819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b"
Sep 17 21:33:43.001505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.001464    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b"} err="failed to get container status \"819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b\": rpc error: code = NotFound desc = could not find container \"819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b\": container with ID starting with 819d1e61eff36bb8f40252b1059caa2a90572a5cc749e84c951431c611515c0b not found: ID does not exist"
Sep 17 21:33:43.001505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.001479    2586 scope.go:117] "RemoveContainer" containerID="01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872"
Sep 17 21:33:43.001728 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:33:43.001711    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872\": container with ID starting with 01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872 not found: ID does not exist" containerID="01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872"
Sep 17 21:33:43.001769 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.001731    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872"} err="failed to get container status \"01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872\": rpc error: code = NotFound desc = could not find container \"01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872\": container with ID starting with 01a297329a1fdd016caa4f9fb3e7e6470dc64918a58b7faacd9f7fa4b889c872 not found: ID does not exist"
Sep 17 21:33:43.640681 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640647    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"]
Sep 17 21:33:43.640907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640894    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="storage-initializer"
Sep 17 21:33:43.640952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640908    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="storage-initializer"
Sep 17 21:33:43.640952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640920    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container"
Sep 17 21:33:43.640952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640926    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container"
Sep 17 21:33:43.640952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640934    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kube-rbac-proxy"
Sep 17 21:33:43.640952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640940    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kube-rbac-proxy"
Sep 17 21:33:43.640952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640948    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kube-rbac-proxy"
Sep 17 21:33:43.640952 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640954    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kube-rbac-proxy"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640966    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="storage-initializer"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640972    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="storage-initializer"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640977    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kserve-container"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640982    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kserve-container"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640987    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kube-rbac-proxy"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.640992    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kube-rbac-proxy"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641001    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="storage-initializer"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641006    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="storage-initializer"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641010    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641017    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641052    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kube-rbac-proxy"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641059    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kube-rbac-proxy"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641066    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c93d49-7ff8-49f4-9931-e12eadb52f96" containerName="kserve-container"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641071    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3c66c93-639d-4a82-a506-1208bc26dd2e" containerName="kserve-container"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641076    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kube-rbac-proxy"
Sep 17 21:33:43.641168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.641081    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" containerName="kserve-container"
Sep 17 21:33:43.645216 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.645198    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.663870 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.663830    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 21:33:43.663990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.663974    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:33:43.664033 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.663987    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 21:33:43.664110 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.664039    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:33:43.665668 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.665651    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:33:43.665775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.665761    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:33:43.669855 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.669841    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:33:43.677327 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.677306    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"]
Sep 17 21:33:43.752577 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.752552    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.752704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.752580    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttmh\" (UniqueName: \"kubernetes.io/projected/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-api-access-kttmh\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.752704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.752606    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.752704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.752691    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ced0b-8422-4284-b408-e45ea01335f4-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.752817 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.752723    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ced0b-8422-4284-b408-e45ea01335f4-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.853229 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.853193    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ced0b-8422-4284-b408-e45ea01335f4-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.853671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.853269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ced0b-8422-4284-b408-e45ea01335f4-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.853671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.853313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.853671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.853340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttmh\" (UniqueName: \"kubernetes.io/projected/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-api-access-kttmh\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.853671 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.853396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.853836 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.853694    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ced0b-8422-4284-b408-e45ea01335f4-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.854152 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.854129    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.854324 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.854302    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.856535 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.856512    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ced0b-8422-4284-b408-e45ea01335f4-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.867608 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.867590    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttmh\" (UniqueName: \"kubernetes.io/projected/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-api-access-kttmh\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:43.954339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:43.954283    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:44.226239 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.226214    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"]
Sep 17 21:33:44.232535 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:33:44.232505    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2ced0b_8422_4284_b408_e45ea01335f4.slice/crio-b567f563f56cde75160f03bf1276136ec460cfd3bb139e3d068d7b470360f2f5 WatchSource:0}: Error finding container b567f563f56cde75160f03bf1276136ec460cfd3bb139e3d068d7b470360f2f5: Status 404 returned error can't find the container with id b567f563f56cde75160f03bf1276136ec460cfd3bb139e3d068d7b470360f2f5
Sep 17 21:33:44.343926 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.343896    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"]
Sep 17 21:33:44.347179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.347163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.349507 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.349483    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 21:33:44.349687 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.349569    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 21:33:44.349815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.349800    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 21:33:44.368849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.368828    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"]
Sep 17 21:33:44.457207 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.457173    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrm6\" (UniqueName: \"kubernetes.io/projected/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-api-access-ljrm6\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.457346 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.457220    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d271fade-02e9-47d5-83da-ef1aab3f581f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.457346 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.457239    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.457346 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.457267    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d271fade-02e9-47d5-83da-ef1aab3f581f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.558749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.558668    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrm6\" (UniqueName: \"kubernetes.io/projected/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-api-access-ljrm6\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.558749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.558734    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d271fade-02e9-47d5-83da-ef1aab3f581f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.558942 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.558766    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.558942 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.558808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d271fade-02e9-47d5-83da-ef1aab3f581f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.559170 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.559139    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d271fade-02e9-47d5-83da-ef1aab3f581f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.559539 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.559517    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.561430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.561408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d271fade-02e9-47d5-83da-ef1aab3f581f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.566823 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.566803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrm6\" (UniqueName: \"kubernetes.io/projected/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-api-access-ljrm6\") pod \"isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.656663 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.656634    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:44.671789 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.671760    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5746400-f112-4ca9-8bae-43494ee73af4" path="/var/lib/kubelet/pods/b5746400-f112-4ca9-8bae-43494ee73af4/volumes"
Sep 17 21:33:44.927870 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.927845    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"]
Sep 17 21:33:44.929706 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:33:44.929681    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd271fade_02e9_47d5_83da_ef1aab3f581f.slice/crio-4e3f92a243f9865d07abfb8ad55db4953e67ec942100ee9698e72e3387721801 WatchSource:0}: Error finding container 4e3f92a243f9865d07abfb8ad55db4953e67ec942100ee9698e72e3387721801: Status 404 returned error can't find the container with id 4e3f92a243f9865d07abfb8ad55db4953e67ec942100ee9698e72e3387721801
Sep 17 21:33:44.985689 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.985659    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerStarted","Data":"4e3f92a243f9865d07abfb8ad55db4953e67ec942100ee9698e72e3387721801"}
Sep 17 21:33:44.987312 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.987284    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerStarted","Data":"6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326"}
Sep 17 21:33:44.987414 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:44.987351    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerStarted","Data":"b567f563f56cde75160f03bf1276136ec460cfd3bb139e3d068d7b470360f2f5"}
Sep 17 21:33:45.992614 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:45.992579    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerStarted","Data":"321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383"}
Sep 17 21:33:46.997245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:46.997157    2586 generic.go:356] "Generic (PLEG): container finished" podID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerID="321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383" exitCode=0
Sep 17 21:33:46.997245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:46.997203    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerDied","Data":"321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383"}
Sep 17 21:33:48.003556 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:48.003517    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerStarted","Data":"00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055"}
Sep 17 21:33:48.004112 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:48.003593    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerStarted","Data":"6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4"}
Sep 17 21:33:48.004112 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:48.003643    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:48.024855 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:48.024810    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podStartSLOduration=2.8751200519999998 podStartE2EDuration="4.024795143s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.149675091s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:33:48.022999444 +0000 UTC m=+2643.556939254" watchObservedRunningTime="2026-09-17 21:33:48.024795143 +0000 UTC m=+2643.558734951"
Sep 17 21:33:49.009075 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:49.009041    2586 generic.go:356] "Generic (PLEG): container finished" podID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerID="6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326" exitCode=0
Sep 17 21:33:49.009528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:49.009111    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerDied","Data":"6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326"}
Sep 17 21:33:49.009528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:49.009487    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:49.010830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:49.010806    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:33:50.017673 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:50.017626    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerStarted","Data":"9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df"}
Sep 17 21:33:50.018178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:50.017705    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerStarted","Data":"c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f"}
Sep 17 21:33:50.018178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:50.017888    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:33:50.018178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:50.018119    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:50.018382 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:50.018270    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:50.019634 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:50.019602    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 21:33:51.022595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:51.022562    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 21:33:55.022653 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:55.022625    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:33:55.023218 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:55.023191    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:33:55.045388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:55.045323    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" podStartSLOduration=8.182801903 podStartE2EDuration="12.045308692s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.862506789s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:33:50.038641347 +0000 UTC m=+2645.572581157" watchObservedRunningTime="2026-09-17 21:33:55.045308692 +0000 UTC m=+2650.579248504"
Sep 17 21:33:56.026687 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:56.026659    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:33:56.027320 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:33:56.027291    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 21:34:05.023637 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:05.023599    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:34:06.028499 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:06.028471    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:34:13.731289 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:13.731257    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"]
Sep 17 21:34:13.731763 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:13.731636    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kserve-container" containerID="cri-o://c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f" gracePeriod=30
Sep 17 21:34:13.731763 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:13.731682    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kube-rbac-proxy" containerID="cri-o://9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df" gracePeriod=30
Sep 17 21:34:14.110758 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.110725    2586 generic.go:356] "Generic (PLEG): container finished" podID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerID="9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df" exitCode=2
Sep 17 21:34:14.110933 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.110774    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerDied","Data":"9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df"}
Sep 17 21:34:14.377374 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.377337    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:34:14.479924 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.479899    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ced0b-8422-4284-b408-e45ea01335f4-proxy-tls\") pod \"2a2ced0b-8422-4284-b408-e45ea01335f4\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") "
Sep 17 21:34:14.480066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.479942    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-rbac-proxy-sar-config\") pod \"2a2ced0b-8422-4284-b408-e45ea01335f4\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") "
Sep 17 21:34:14.480066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.479966    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kttmh\" (UniqueName: \"kubernetes.io/projected/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-api-access-kttmh\") pod \"2a2ced0b-8422-4284-b408-e45ea01335f4\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") "
Sep 17 21:34:14.480066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.480007    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-cabundle-cert\") pod \"2a2ced0b-8422-4284-b408-e45ea01335f4\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") "
Sep 17 21:34:14.480066 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.480042    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ced0b-8422-4284-b408-e45ea01335f4-kserve-provision-location\") pod \"2a2ced0b-8422-4284-b408-e45ea01335f4\" (UID: \"2a2ced0b-8422-4284-b408-e45ea01335f4\") "
Sep 17 21:34:14.480390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.480336    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a2ced0b-8422-4284-b408-e45ea01335f4" (UID: "2a2ced0b-8422-4284-b408-e45ea01335f4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:34:14.480390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.480297    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a2ced0b-8422-4284-b408-e45ea01335f4" (UID: "2a2ced0b-8422-4284-b408-e45ea01335f4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:34:14.480536 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.480476    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2ced0b-8422-4284-b408-e45ea01335f4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a2ced0b-8422-4284-b408-e45ea01335f4" (UID: "2a2ced0b-8422-4284-b408-e45ea01335f4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:34:14.481981 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.481950    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-api-access-kttmh" (OuterVolumeSpecName: "kube-api-access-kttmh") pod "2a2ced0b-8422-4284-b408-e45ea01335f4" (UID: "2a2ced0b-8422-4284-b408-e45ea01335f4"). InnerVolumeSpecName "kube-api-access-kttmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:34:14.482138 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.482118    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2ced0b-8422-4284-b408-e45ea01335f4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a2ced0b-8422-4284-b408-e45ea01335f4" (UID: "2a2ced0b-8422-4284-b408-e45ea01335f4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:34:14.580773 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.580743    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a2ced0b-8422-4284-b408-e45ea01335f4-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:34:14.580773 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.580769    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:34:14.580773 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.580779    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttmh\" (UniqueName: \"kubernetes.io/projected/2a2ced0b-8422-4284-b408-e45ea01335f4-kube-api-access-kttmh\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:34:14.580960 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.580787    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a2ced0b-8422-4284-b408-e45ea01335f4-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:34:14.580960 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:14.580796    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a2ced0b-8422-4284-b408-e45ea01335f4-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:34:15.023466 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.023381    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:34:15.116730 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.116700    2586 generic.go:356] "Generic (PLEG): container finished" podID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerID="c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f" exitCode=0
Sep 17 21:34:15.116882 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.116739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerDied","Data":"c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f"}
Sep 17 21:34:15.116882 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.116753    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"
Sep 17 21:34:15.116882 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.116797    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb" event={"ID":"2a2ced0b-8422-4284-b408-e45ea01335f4","Type":"ContainerDied","Data":"b567f563f56cde75160f03bf1276136ec460cfd3bb139e3d068d7b470360f2f5"}
Sep 17 21:34:15.116882 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.116832    2586 scope.go:117] "RemoveContainer" containerID="9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df"
Sep 17 21:34:15.124727 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.124711    2586 scope.go:117] "RemoveContainer" containerID="c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f"
Sep 17 21:34:15.131691 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.131671    2586 scope.go:117] "RemoveContainer" containerID="6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326"
Sep 17 21:34:15.138621 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.138604    2586 scope.go:117] "RemoveContainer" containerID="9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df"
Sep 17 21:34:15.138860 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:34:15.138844    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df\": container with ID starting with 9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df not found: ID does not exist" containerID="9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df"
Sep 17 21:34:15.138903 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.138865    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df"} err="failed to get container status \"9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df\": rpc error: code = NotFound desc = could not find container \"9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df\": container with ID starting with 9d59375484ec3af3e453cd1d0ccc6702b0a5bd3552823da2a0a22436806dd3df not found: ID does not exist"
Sep 17 21:34:15.138903 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.138881    2586 scope.go:117] "RemoveContainer" containerID="c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f"
Sep 17 21:34:15.139121 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:34:15.139101    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f\": container with ID starting with c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f not found: ID does not exist" containerID="c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f"
Sep 17 21:34:15.139164 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.139126    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f"} err="failed to get container status \"c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f\": rpc error: code = NotFound desc = could not find container \"c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f\": container with ID starting with c38039bc1e36ebc273bfa511baa02e99e8c978cee8cd4906eccda79448afa01f not found: ID does not exist"
Sep 17 21:34:15.139164 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.139140    2586 scope.go:117] "RemoveContainer" containerID="6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326"
Sep 17 21:34:15.139365 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:34:15.139338    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326\": container with ID starting with 6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326 not found: ID does not exist" containerID="6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326"
Sep 17 21:34:15.139416 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.139376    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326"} err="failed to get container status \"6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326\": rpc error: code = NotFound desc = could not find container \"6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326\": container with ID starting with 6de58aa8dc1019e202a179a844443b76607c7f65cf27575d70dd9ea81bf5b326 not found: ID does not exist"
Sep 17 21:34:15.147420 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.147398    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"]
Sep 17 21:34:15.152477 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:15.152458    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-rwjkb"]
Sep 17 21:34:16.670500 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:16.670464    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" path="/var/lib/kubelet/pods/2a2ced0b-8422-4284-b408-e45ea01335f4/volumes"
Sep 17 21:34:19.009823 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.009792    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"]
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010033    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kube-rbac-proxy"
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010044    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kube-rbac-proxy"
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010057    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="storage-initializer"
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010062    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="storage-initializer"
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010072    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kserve-container"
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010078    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kserve-container"
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010137    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kserve-container"
Sep 17 21:34:19.010183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.010143    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2ced0b-8422-4284-b408-e45ea01335f4" containerName="kube-rbac-proxy"
Sep 17 21:34:19.013170 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.013148    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.015684 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.015666    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 21:34:19.015789 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.015711    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:34:19.015991 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.015975    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:34:19.016076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.015995    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:34:19.023899 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.023878    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"]
Sep 17 21:34:19.108513 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.108489    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a05272a-777d-46de-b1b1-2a089113bb44-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.108622 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.108515    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a05272a-777d-46de-b1b1-2a089113bb44-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.108622 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.108542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.108622 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.108560    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.108622 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.108609    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/9a05272a-777d-46de-b1b1-2a089113bb44-kube-api-access-mxcm2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.208930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.208902    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.209036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.208935    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.209036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.208974    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/9a05272a-777d-46de-b1b1-2a089113bb44-kube-api-access-mxcm2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.209036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.209024    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a05272a-777d-46de-b1b1-2a089113bb44-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.209195 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.209049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a05272a-777d-46de-b1b1-2a089113bb44-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.209444 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.209428    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a05272a-777d-46de-b1b1-2a089113bb44-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.209738 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.209718    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.209784 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.209718    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.211396 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.211373    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a05272a-777d-46de-b1b1-2a089113bb44-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.217477 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.217457    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/9a05272a-777d-46de-b1b1-2a089113bb44-kube-api-access-mxcm2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.323182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.323119    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:19.595503 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:19.595479    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"]
Sep 17 21:34:19.597850 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:34:19.597822    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a05272a_777d_46de_b1b1_2a089113bb44.slice/crio-13c01b19b212dcaa472434a0c3a29dcf516681185990c7f5d7fe9d8c40b09b82 WatchSource:0}: Error finding container 13c01b19b212dcaa472434a0c3a29dcf516681185990c7f5d7fe9d8c40b09b82: Status 404 returned error can't find the container with id 13c01b19b212dcaa472434a0c3a29dcf516681185990c7f5d7fe9d8c40b09b82
Sep 17 21:34:20.136757 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:20.136721    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerStarted","Data":"6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d"}
Sep 17 21:34:20.137275 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:20.136783    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerStarted","Data":"13c01b19b212dcaa472434a0c3a29dcf516681185990c7f5d7fe9d8c40b09b82"}
Sep 17 21:34:24.151774 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:24.151741    2586 generic.go:356] "Generic (PLEG): container finished" podID="9a05272a-777d-46de-b1b1-2a089113bb44" containerID="6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d" exitCode=0
Sep 17 21:34:24.152159 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:24.151815    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerDied","Data":"6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d"}
Sep 17 21:34:25.023403 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:25.023347    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:34:25.157544 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:25.157511    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerStarted","Data":"eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f"}
Sep 17 21:34:25.157918 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:25.157572    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerStarted","Data":"9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a"}
Sep 17 21:34:25.157918 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:25.157885    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:25.157997 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:25.157948    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:25.184154 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:25.184107    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" podStartSLOduration=3.10981556 podStartE2EDuration="7.184084347s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.074268787s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:34:25.183044545 +0000 UTC m=+2680.716984355" watchObservedRunningTime="2026-09-17 21:34:25.184084347 +0000 UTC m=+2680.718024156"
Sep 17 21:34:31.170174 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:31.170094    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:34:35.023918 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:35.023872    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:34:45.023338 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:45.023295    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:34:55.023316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:34:55.023275    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 21:35:01.173722 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:01.173692    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:35:04.670849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:04.670823    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:35:09.074073 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:09.074038    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"]
Sep 17 21:35:09.074601 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:09.074376    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kserve-container" containerID="cri-o://9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a" gracePeriod=30
Sep 17 21:35:09.074601 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:09.074435    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kube-rbac-proxy" containerID="cri-o://eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f" gracePeriod=30
Sep 17 21:35:09.321039 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:09.321008    2586 generic.go:356] "Generic (PLEG): container finished" podID="9a05272a-777d-46de-b1b1-2a089113bb44" containerID="eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f" exitCode=2
Sep 17 21:35:09.321227 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:09.321078    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerDied","Data":"eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f"}
Sep 17 21:35:10.138078 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.138056    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:35:10.165100 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165067    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a05272a-777d-46de-b1b1-2a089113bb44-proxy-tls\") pod \"9a05272a-777d-46de-b1b1-2a089113bb44\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") "
Sep 17 21:35:10.165206 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165107    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-kube-rbac-proxy-sar-config\") pod \"9a05272a-777d-46de-b1b1-2a089113bb44\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") "
Sep 17 21:35:10.165206 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165139    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/9a05272a-777d-46de-b1b1-2a089113bb44-kube-api-access-mxcm2\") pod \"9a05272a-777d-46de-b1b1-2a089113bb44\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") "
Sep 17 21:35:10.165206 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165161    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-cabundle-cert\") pod \"9a05272a-777d-46de-b1b1-2a089113bb44\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") "
Sep 17 21:35:10.165206 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165193    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a05272a-777d-46de-b1b1-2a089113bb44-kserve-provision-location\") pod \"9a05272a-777d-46de-b1b1-2a089113bb44\" (UID: \"9a05272a-777d-46de-b1b1-2a089113bb44\") "
Sep 17 21:35:10.165543 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165506    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a05272a-777d-46de-b1b1-2a089113bb44" (UID: "9a05272a-777d-46de-b1b1-2a089113bb44"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:35:10.165665 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165591    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a05272a-777d-46de-b1b1-2a089113bb44" (UID: "9a05272a-777d-46de-b1b1-2a089113bb44"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:35:10.165665 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.165654    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a05272a-777d-46de-b1b1-2a089113bb44-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a05272a-777d-46de-b1b1-2a089113bb44" (UID: "9a05272a-777d-46de-b1b1-2a089113bb44"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:35:10.167146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.167119    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a05272a-777d-46de-b1b1-2a089113bb44-kube-api-access-mxcm2" (OuterVolumeSpecName: "kube-api-access-mxcm2") pod "9a05272a-777d-46de-b1b1-2a089113bb44" (UID: "9a05272a-777d-46de-b1b1-2a089113bb44"). InnerVolumeSpecName "kube-api-access-mxcm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:35:10.167476 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.167453    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a05272a-777d-46de-b1b1-2a089113bb44-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a05272a-777d-46de-b1b1-2a089113bb44" (UID: "9a05272a-777d-46de-b1b1-2a089113bb44"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:35:10.266645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.266591    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a05272a-777d-46de-b1b1-2a089113bb44-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:10.266645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.266611    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:10.266645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.266621    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxcm2\" (UniqueName: \"kubernetes.io/projected/9a05272a-777d-46de-b1b1-2a089113bb44-kube-api-access-mxcm2\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:10.266645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.266630    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a05272a-777d-46de-b1b1-2a089113bb44-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:10.266645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.266638    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a05272a-777d-46de-b1b1-2a089113bb44-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:10.327127 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.327101    2586 generic.go:356] "Generic (PLEG): container finished" podID="9a05272a-777d-46de-b1b1-2a089113bb44" containerID="9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a" exitCode=0
Sep 17 21:35:10.327262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.327146    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerDied","Data":"9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a"}
Sep 17 21:35:10.327262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.327158    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"
Sep 17 21:35:10.327262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.327221    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5" event={"ID":"9a05272a-777d-46de-b1b1-2a089113bb44","Type":"ContainerDied","Data":"13c01b19b212dcaa472434a0c3a29dcf516681185990c7f5d7fe9d8c40b09b82"}
Sep 17 21:35:10.327262 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.327247    2586 scope.go:117] "RemoveContainer" containerID="eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f"
Sep 17 21:35:10.337322 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.337307    2586 scope.go:117] "RemoveContainer" containerID="9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a"
Sep 17 21:35:10.344712 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.344690    2586 scope.go:117] "RemoveContainer" containerID="6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d"
Sep 17 21:35:10.351107 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.351091    2586 scope.go:117] "RemoveContainer" containerID="eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f"
Sep 17 21:35:10.351347 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:35:10.351328    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f\": container with ID starting with eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f not found: ID does not exist" containerID="eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f"
Sep 17 21:35:10.351430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.351367    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f"} err="failed to get container status \"eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f\": rpc error: code = NotFound desc = could not find container \"eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f\": container with ID starting with eaaf08871ea29d405f12272cbe10f7f53fdbcf55c815c53bf96c6e4ab7a3679f not found: ID does not exist"
Sep 17 21:35:10.351430 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.351385    2586 scope.go:117] "RemoveContainer" containerID="9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a"
Sep 17 21:35:10.351622 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:35:10.351604    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a\": container with ID starting with 9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a not found: ID does not exist" containerID="9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a"
Sep 17 21:35:10.351661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.351628    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a"} err="failed to get container status \"9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a\": rpc error: code = NotFound desc = could not find container \"9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a\": container with ID starting with 9400302f3063c44a2177105ed90442352cebc454d7e4e63df6581aacf848685a not found: ID does not exist"
Sep 17 21:35:10.351661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.351643    2586 scope.go:117] "RemoveContainer" containerID="6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d"
Sep 17 21:35:10.351878 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:35:10.351863    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d\": container with ID starting with 6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d not found: ID does not exist" containerID="6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d"
Sep 17 21:35:10.351930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.351881    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d"} err="failed to get container status \"6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d\": rpc error: code = NotFound desc = could not find container \"6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d\": container with ID starting with 6e950de70a15235aae3e127f1e92d493aa7cd5cb6258b7e5bcafcdfeb77a876d not found: ID does not exist"
Sep 17 21:35:10.356017 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.355997    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"]
Sep 17 21:35:10.360930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.360910    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-v8lr5"]
Sep 17 21:35:10.670997 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:10.670964    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" path="/var/lib/kubelet/pods/9a05272a-777d-46de-b1b1-2a089113bb44/volumes"
Sep 17 21:35:14.336181 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336151    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"]
Sep 17 21:35:14.336660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336534    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kserve-container"
Sep 17 21:35:14.336660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336552    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kserve-container"
Sep 17 21:35:14.336660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336570    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kube-rbac-proxy"
Sep 17 21:35:14.336660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336578    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kube-rbac-proxy"
Sep 17 21:35:14.336660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336596    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="storage-initializer"
Sep 17 21:35:14.336660 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336606    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="storage-initializer"
Sep 17 21:35:14.336978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336666    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kserve-container"
Sep 17 21:35:14.336978 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.336678    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a05272a-777d-46de-b1b1-2a089113bb44" containerName="kube-rbac-proxy"
Sep 17 21:35:14.341399 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.341375    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.344956 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.344932    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 21:35:14.345406 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.345391    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:35:14.345528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.345511    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:35:14.345604 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.345526    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 21:35:14.379738 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.379712    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"]
Sep 17 21:35:14.393834 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.393808    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsgzm\" (UniqueName: \"kubernetes.io/projected/357220d8-850d-4583-830f-a17aafcab5a7-kube-api-access-xsgzm\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.393958 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.393860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357220d8-850d-4583-830f-a17aafcab5a7-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.393958 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.393888    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.393958 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.393914    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357220d8-850d-4583-830f-a17aafcab5a7-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.393958 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.393949    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.494654 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.494631    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsgzm\" (UniqueName: \"kubernetes.io/projected/357220d8-850d-4583-830f-a17aafcab5a7-kube-api-access-xsgzm\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.494775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.494669    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357220d8-850d-4583-830f-a17aafcab5a7-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.494775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.494690    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.494775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.494708    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357220d8-850d-4583-830f-a17aafcab5a7-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.494775 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.494733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.495240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.495209    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357220d8-850d-4583-830f-a17aafcab5a7-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.495420 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.495401    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.495498 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.495455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.497089 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.497064    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357220d8-850d-4583-830f-a17aafcab5a7-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.503609 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.503587    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsgzm\" (UniqueName: \"kubernetes.io/projected/357220d8-850d-4583-830f-a17aafcab5a7-kube-api-access-xsgzm\") pod \"isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.651694 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.651667    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:14.960436 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:14.960332    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"]
Sep 17 21:35:14.964332 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:35:14.964291    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357220d8_850d_4583_830f_a17aafcab5a7.slice/crio-028c10a2c60a434b233382db55f551189416632891d9bb7d8e0613ee69757d88 WatchSource:0}: Error finding container 028c10a2c60a434b233382db55f551189416632891d9bb7d8e0613ee69757d88: Status 404 returned error can't find the container with id 028c10a2c60a434b233382db55f551189416632891d9bb7d8e0613ee69757d88
Sep 17 21:35:15.027933 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.027904    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"]
Sep 17 21:35:15.028138 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.028115    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container" containerID="cri-o://6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4" gracePeriod=30
Sep 17 21:35:15.028239 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.028163    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kube-rbac-proxy" containerID="cri-o://00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055" gracePeriod=30
Sep 17 21:35:15.176008 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.175976    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"]
Sep 17 21:35:15.179200 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.179183    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.182162 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.182143    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 21:35:15.182162 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.182151    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 21:35:15.198544 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.198521    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"]
Sep 17 21:35:15.301040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.300955    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.301040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.301001    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903191b-4ed6-4a42-817d-39ffd25c5791-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.301040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.301028    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhn87\" (UniqueName: \"kubernetes.io/projected/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-api-access-dhn87\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.301299 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.301098    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a903191b-4ed6-4a42-817d-39ffd25c5791-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.301299 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.301146    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.349267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.349237    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerStarted","Data":"a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2"}
Sep 17 21:35:15.349672 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.349309    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerStarted","Data":"028c10a2c60a434b233382db55f551189416632891d9bb7d8e0613ee69757d88"}
Sep 17 21:35:15.351681 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.351651    2586 generic.go:356] "Generic (PLEG): container finished" podID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerID="00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055" exitCode=2
Sep 17 21:35:15.351791 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.351721    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerDied","Data":"00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055"}
Sep 17 21:35:15.401670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.401642    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903191b-4ed6-4a42-817d-39ffd25c5791-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.401804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.401688    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhn87\" (UniqueName: \"kubernetes.io/projected/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-api-access-dhn87\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.401804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.401722    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a903191b-4ed6-4a42-817d-39ffd25c5791-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.401804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.401752    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.401983 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.401823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.402201 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.402153    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a903191b-4ed6-4a42-817d-39ffd25c5791-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.402687 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.402661    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.402767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.402670    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.404115 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.404090    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903191b-4ed6-4a42-817d-39ffd25c5791-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.410313 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.410295    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhn87\" (UniqueName: \"kubernetes.io/projected/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-api-access-dhn87\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.489506 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.489474    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:15.764406 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:15.764196    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"]
Sep 17 21:35:15.766696 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:35:15.766669    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda903191b_4ed6_4a42_817d_39ffd25c5791.slice/crio-8e79197515cd62252527440b5f131e95424b8f54c35b22a8c6d4765a275e2b7b WatchSource:0}: Error finding container 8e79197515cd62252527440b5f131e95424b8f54c35b22a8c6d4765a275e2b7b: Status 404 returned error can't find the container with id 8e79197515cd62252527440b5f131e95424b8f54c35b22a8c6d4765a275e2b7b
Sep 17 21:35:16.357432 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:16.357399    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerStarted","Data":"b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69"}
Sep 17 21:35:16.357820 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:16.357452    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerStarted","Data":"8e79197515cd62252527440b5f131e95424b8f54c35b22a8c6d4765a275e2b7b"}
Sep 17 21:35:17.362454 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:17.362420    2586 generic.go:356] "Generic (PLEG): container finished" podID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerID="b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69" exitCode=0
Sep 17 21:35:17.362811 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:17.362462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerDied","Data":"b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69"}
Sep 17 21:35:18.367889 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:18.367857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerStarted","Data":"a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e"}
Sep 17 21:35:18.368245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:18.367910    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerStarted","Data":"5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e"}
Sep 17 21:35:18.368245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:18.368062    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:18.368245 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:18.368205    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:18.369406 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:18.369382    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:35:18.393850 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:18.393809    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podStartSLOduration=2.381052149 podStartE2EDuration="3.393797254s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.012745105s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:35:18.392426301 +0000 UTC m=+2733.926366112" watchObservedRunningTime="2026-09-17 21:35:18.393797254 +0000 UTC m=+2733.927737062"
Sep 17 21:35:19.373878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.373844    2586 generic.go:356] "Generic (PLEG): container finished" podID="357220d8-850d-4583-830f-a17aafcab5a7" containerID="a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2" exitCode=0
Sep 17 21:35:19.374230 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.373921    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerDied","Data":"a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2"}
Sep 17 21:35:19.374230 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.374198    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:35:19.428279 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.428263    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:35:19.665551 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.665528    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:35:19.734619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.734593    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d271fade-02e9-47d5-83da-ef1aab3f581f-kserve-provision-location\") pod \"d271fade-02e9-47d5-83da-ef1aab3f581f\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") "
Sep 17 21:35:19.734757 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.734626    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-rbac-proxy-sar-config\") pod \"d271fade-02e9-47d5-83da-ef1aab3f581f\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") "
Sep 17 21:35:19.734757 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.734664    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljrm6\" (UniqueName: \"kubernetes.io/projected/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-api-access-ljrm6\") pod \"d271fade-02e9-47d5-83da-ef1aab3f581f\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") "
Sep 17 21:35:19.734757 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.734702    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d271fade-02e9-47d5-83da-ef1aab3f581f-proxy-tls\") pod \"d271fade-02e9-47d5-83da-ef1aab3f581f\" (UID: \"d271fade-02e9-47d5-83da-ef1aab3f581f\") "
Sep 17 21:35:19.734960 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.734941    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d271fade-02e9-47d5-83da-ef1aab3f581f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d271fade-02e9-47d5-83da-ef1aab3f581f" (UID: "d271fade-02e9-47d5-83da-ef1aab3f581f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:35:19.735061 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.735030    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d271fade-02e9-47d5-83da-ef1aab3f581f" (UID: "d271fade-02e9-47d5-83da-ef1aab3f581f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:35:19.736676 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.736649    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d271fade-02e9-47d5-83da-ef1aab3f581f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d271fade-02e9-47d5-83da-ef1aab3f581f" (UID: "d271fade-02e9-47d5-83da-ef1aab3f581f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:35:19.736753 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.736691    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-api-access-ljrm6" (OuterVolumeSpecName: "kube-api-access-ljrm6") pod "d271fade-02e9-47d5-83da-ef1aab3f581f" (UID: "d271fade-02e9-47d5-83da-ef1aab3f581f"). InnerVolumeSpecName "kube-api-access-ljrm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:35:19.835691 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.835662    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrm6\" (UniqueName: \"kubernetes.io/projected/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-api-access-ljrm6\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:19.835691 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.835690    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d271fade-02e9-47d5-83da-ef1aab3f581f-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:19.835850 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.835703    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d271fade-02e9-47d5-83da-ef1aab3f581f-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:19.835850 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:19.835712    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d271fade-02e9-47d5-83da-ef1aab3f581f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:35:20.381087 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.381050    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerStarted","Data":"0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325"}
Sep 17 21:35:20.384086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.383789    2586 generic.go:356] "Generic (PLEG): container finished" podID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerID="6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4" exitCode=0
Sep 17 21:35:20.384086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.383852    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerDied","Data":"6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4"}
Sep 17 21:35:20.384086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.383864    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"
Sep 17 21:35:20.384086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.383909    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw" event={"ID":"d271fade-02e9-47d5-83da-ef1aab3f581f","Type":"ContainerDied","Data":"4e3f92a243f9865d07abfb8ad55db4953e67ec942100ee9698e72e3387721801"}
Sep 17 21:35:20.384086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.383945    2586 scope.go:117] "RemoveContainer" containerID="00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055"
Sep 17 21:35:20.395915 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.395893    2586 scope.go:117] "RemoveContainer" containerID="6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4"
Sep 17 21:35:20.405594 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.405576    2586 scope.go:117] "RemoveContainer" containerID="321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383"
Sep 17 21:35:20.417472 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.417381    2586 scope.go:117] "RemoveContainer" containerID="00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055"
Sep 17 21:35:20.417744 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:35:20.417718    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055\": container with ID starting with 00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055 not found: ID does not exist" containerID="00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055"
Sep 17 21:35:20.417840 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.417756    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055"} err="failed to get container status \"00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055\": rpc error: code = NotFound desc = could not find container \"00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055\": container with ID starting with 00343c5533c3ece40c6748daa821b2caea736cf379c95e050c7ee7cb34b96055 not found: ID does not exist"
Sep 17 21:35:20.417840 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.417781    2586 scope.go:117] "RemoveContainer" containerID="6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4"
Sep 17 21:35:20.418743 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:35:20.418339    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4\": container with ID starting with 6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4 not found: ID does not exist" containerID="6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4"
Sep 17 21:35:20.418743 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.418516    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4"} err="failed to get container status \"6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4\": rpc error: code = NotFound desc = could not find container \"6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4\": container with ID starting with 6713bba311df695619e71e566b442faae4f7d4cbb65f4d6116cc3e12f854acd4 not found: ID does not exist"
Sep 17 21:35:20.418743 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.418538    2586 scope.go:117] "RemoveContainer" containerID="321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383"
Sep 17 21:35:20.419319 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:35:20.419237    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383\": container with ID starting with 321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383 not found: ID does not exist" containerID="321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383"
Sep 17 21:35:20.419319 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.419269    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383"} err="failed to get container status \"321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383\": rpc error: code = NotFound desc = could not find container \"321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383\": container with ID starting with 321c072b8b2fa8768420fbbfdfa360b35e8917c6ec614dc3d147a0fffb33b383 not found: ID does not exist"
Sep 17 21:35:20.423162 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.423089    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"]
Sep 17 21:35:20.431091 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.431032    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6b6db9fddf-mrmpw"]
Sep 17 21:35:20.673384 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:20.673279    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" path="/var/lib/kubelet/pods/d271fade-02e9-47d5-83da-ef1aab3f581f/volumes"
Sep 17 21:35:22.395822 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:22.395784    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerStarted","Data":"074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0"}
Sep 17 21:35:22.396193 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:22.395884    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerStarted","Data":"d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a"}
Sep 17 21:35:22.396193 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:22.395923    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:22.396193 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:22.395948    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:22.434806 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:22.434758    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podStartSLOduration=1.651809433 podStartE2EDuration="8.434741131s" totalImagesPullingTime="2.811661595s" totalInitContainerRuntime="3.971270103s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:35:22.434651294 +0000 UTC m=+2737.968591139" watchObservedRunningTime="2026-09-17 21:35:22.434741131 +0000 UTC m=+2737.968680940"
Sep 17 21:35:23.400506 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:23.400470    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:24.378705 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:24.378677    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:35:24.379149 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:24.379124    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:35:29.409711 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:29.409678    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:35:34.379974 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:34.379931    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:35:44.379152 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:44.379111    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:35:49.411255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:49.411214    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 21:35:52.508693 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:52.508652    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:35:52.511102 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:52.511077    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:35:54.379876 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:54.379837    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:35:54.652109 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:54.652025    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 21:35:59.622000 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:35:59.621906    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:36:04.379711 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:04.379673    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:36:14.379733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:14.379694    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:36:24.379582 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:24.379535    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 21:36:29.623519 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:29.623486    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:36:34.380131 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.380031    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"]
Sep 17 21:36:34.381558 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.380166    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:36:34.381558 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.380581    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container" containerID="cri-o://0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325" gracePeriod=30
Sep 17 21:36:34.381558 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.380602    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" containerID="cri-o://074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0" gracePeriod=30
Sep 17 21:36:34.381558 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.380608    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-agent" containerID="cri-o://d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a" gracePeriod=30
Sep 17 21:36:34.405541 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.405513    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 21:36:34.664141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.664055    2586 generic.go:356] "Generic (PLEG): container finished" podID="357220d8-850d-4583-830f-a17aafcab5a7" containerID="074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0" exitCode=2
Sep 17 21:36:34.664141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:34.664132    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerDied","Data":"074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0"}
Sep 17 21:36:35.121193 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:35.121157    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"]
Sep 17 21:36:35.121432 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:35.121409    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container" containerID="cri-o://5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e" gracePeriod=30
Sep 17 21:36:35.121509 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:35.121435    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kube-rbac-proxy" containerID="cri-o://a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e" gracePeriod=30
Sep 17 21:36:35.669858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:35.669825    2586 generic.go:356] "Generic (PLEG): container finished" podID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerID="a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e" exitCode=2
Sep 17 21:36:35.670198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:35.669897    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerDied","Data":"a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e"}
Sep 17 21:36:36.675431 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:36.675403    2586 generic.go:356] "Generic (PLEG): container finished" podID="357220d8-850d-4583-830f-a17aafcab5a7" containerID="0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325" exitCode=0
Sep 17 21:36:36.675819 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:36.675474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerDied","Data":"0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325"}
Sep 17 21:36:39.375367 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.375322    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 17 21:36:39.405564 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.405527    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 21:36:39.410919 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.410889    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 21:36:39.534717 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.534689    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"]
Sep 17 21:36:39.535004 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.534990    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kube-rbac-proxy"
Sep 17 21:36:39.535004 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.535005    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kube-rbac-proxy"
Sep 17 21:36:39.535171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.535015    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="storage-initializer"
Sep 17 21:36:39.535171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.535021    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="storage-initializer"
Sep 17 21:36:39.535171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.535032    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container"
Sep 17 21:36:39.535171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.535041    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container"
Sep 17 21:36:39.535171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.535092    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kube-rbac-proxy"
Sep 17 21:36:39.535171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.535101    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d271fade-02e9-47d5-83da-ef1aab3f581f" containerName="kserve-container"
Sep 17 21:36:39.537930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.537909    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.540800 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.540775    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 21:36:39.541176 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.541161    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 21:36:39.548839 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.548815    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"]
Sep 17 21:36:39.560626 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.560606    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:36:39.615453 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615416    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-rbac-proxy-sar-config\") pod \"a903191b-4ed6-4a42-817d-39ffd25c5791\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") "
Sep 17 21:36:39.615670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615474    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903191b-4ed6-4a42-817d-39ffd25c5791-proxy-tls\") pod \"a903191b-4ed6-4a42-817d-39ffd25c5791\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") "
Sep 17 21:36:39.615670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615505    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-cabundle-cert\") pod \"a903191b-4ed6-4a42-817d-39ffd25c5791\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") "
Sep 17 21:36:39.615670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615532    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a903191b-4ed6-4a42-817d-39ffd25c5791-kserve-provision-location\") pod \"a903191b-4ed6-4a42-817d-39ffd25c5791\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") "
Sep 17 21:36:39.615670 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615558    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhn87\" (UniqueName: \"kubernetes.io/projected/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-api-access-dhn87\") pod \"a903191b-4ed6-4a42-817d-39ffd25c5791\" (UID: \"a903191b-4ed6-4a42-817d-39ffd25c5791\") "
Sep 17 21:36:39.615907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615685    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fd0782-562c-435d-805c-067839ac18e7-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.615907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615703    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a903191b-4ed6-4a42-817d-39ffd25c5791" (UID: "a903191b-4ed6-4a42-817d-39ffd25c5791"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:36:39.615907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615737    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.615907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615849    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fd0782-562c-435d-805c-067839ac18e7-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.615907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615872    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.615907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615869    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a903191b-4ed6-4a42-817d-39ffd25c5791-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a903191b-4ed6-4a42-817d-39ffd25c5791" (UID: "a903191b-4ed6-4a42-817d-39ffd25c5791"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:36:39.616133 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615912    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a903191b-4ed6-4a42-817d-39ffd25c5791" (UID: "a903191b-4ed6-4a42-817d-39ffd25c5791"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:36:39.616133 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615943    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wkr\" (UniqueName: \"kubernetes.io/projected/27fd0782-562c-435d-805c-067839ac18e7-kube-api-access-v6wkr\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.616133 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.615991    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.616133 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.616006    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a903191b-4ed6-4a42-817d-39ffd25c5791-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.616133 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.616030    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.617600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.617577    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a903191b-4ed6-4a42-817d-39ffd25c5791-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a903191b-4ed6-4a42-817d-39ffd25c5791" (UID: "a903191b-4ed6-4a42-817d-39ffd25c5791"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:36:39.617830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.617763    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-api-access-dhn87" (OuterVolumeSpecName: "kube-api-access-dhn87") pod "a903191b-4ed6-4a42-817d-39ffd25c5791" (UID: "a903191b-4ed6-4a42-817d-39ffd25c5791"). InnerVolumeSpecName "kube-api-access-dhn87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:36:39.687749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.687681    2586 generic.go:356] "Generic (PLEG): container finished" podID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerID="5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e" exitCode=0
Sep 17 21:36:39.687749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.687745    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"
Sep 17 21:36:39.687908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.687765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerDied","Data":"5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e"}
Sep 17 21:36:39.687908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.687817    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9" event={"ID":"a903191b-4ed6-4a42-817d-39ffd25c5791","Type":"ContainerDied","Data":"8e79197515cd62252527440b5f131e95424b8f54c35b22a8c6d4765a275e2b7b"}
Sep 17 21:36:39.687908 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.687843    2586 scope.go:117] "RemoveContainer" containerID="a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e"
Sep 17 21:36:39.696039 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.696023    2586 scope.go:117] "RemoveContainer" containerID="5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e"
Sep 17 21:36:39.703027 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.703011    2586 scope.go:117] "RemoveContainer" containerID="b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69"
Sep 17 21:36:39.709830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.709808    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"]
Sep 17 21:36:39.710815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.710798    2586 scope.go:117] "RemoveContainer" containerID="a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e"
Sep 17 21:36:39.711077 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:36:39.711056    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e\": container with ID starting with a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e not found: ID does not exist" containerID="a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e"
Sep 17 21:36:39.711138 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.711083    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e"} err="failed to get container status \"a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e\": rpc error: code = NotFound desc = could not find container \"a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e\": container with ID starting with a1f2976e078830d42a703fcd6fb042004c18253881e49f54ff0a4cc597dfdf9e not found: ID does not exist"
Sep 17 21:36:39.711138 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.711099    2586 scope.go:117] "RemoveContainer" containerID="5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e"
Sep 17 21:36:39.711331 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:36:39.711316    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e\": container with ID starting with 5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e not found: ID does not exist" containerID="5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e"
Sep 17 21:36:39.711389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.711335    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e"} err="failed to get container status \"5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e\": rpc error: code = NotFound desc = could not find container \"5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e\": container with ID starting with 5bee49dde1d28c899f07b88dbd7938fc617b869cd0a05f2a9dd0ebac8089328e not found: ID does not exist"
Sep 17 21:36:39.711389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.711349    2586 scope.go:117] "RemoveContainer" containerID="b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69"
Sep 17 21:36:39.711597 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:36:39.711581    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69\": container with ID starting with b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69 not found: ID does not exist" containerID="b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69"
Sep 17 21:36:39.711648 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.711600    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69"} err="failed to get container status \"b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69\": rpc error: code = NotFound desc = could not find container \"b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69\": container with ID starting with b6fa5c45de15cb7140cd0a6a7a013ff2f0224666eb76c81378e5a50bcfbb9a69 not found: ID does not exist"
Sep 17 21:36:39.715223 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.715204    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-tl4m9"]
Sep 17 21:36:39.717010 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.716992    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fd0782-562c-435d-805c-067839ac18e7-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.717069 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717029    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.717106 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717064    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fd0782-562c-435d-805c-067839ac18e7-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.717106 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717087    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.717171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717137    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wkr\" (UniqueName: \"kubernetes.io/projected/27fd0782-562c-435d-805c-067839ac18e7-kube-api-access-v6wkr\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.717211 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717178    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a903191b-4ed6-4a42-817d-39ffd25c5791-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.717211 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717193    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhn87\" (UniqueName: \"kubernetes.io/projected/a903191b-4ed6-4a42-817d-39ffd25c5791-kube-api-access-dhn87\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:36:39.717593 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717569    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fd0782-562c-435d-805c-067839ac18e7-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.717764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717746    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.717815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.717792    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.719447 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.719432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fd0782-562c-435d-805c-067839ac18e7-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.725804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.725776    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wkr\" (UniqueName: \"kubernetes.io/projected/27fd0782-562c-435d-805c-067839ac18e7-kube-api-access-v6wkr\") pod \"isvc-paddle-predictor-6d5c9fd4f6-mdqtk\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:39.848538 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:39.848507    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:40.117930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:40.117902    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"]
Sep 17 21:36:40.121030 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:36:40.120999    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fd0782_562c_435d_805c_067839ac18e7.slice/crio-969e830b9035329126e78c6334a06a4dd154211e8015a67d3fd5fd376085817a WatchSource:0}: Error finding container 969e830b9035329126e78c6334a06a4dd154211e8015a67d3fd5fd376085817a: Status 404 returned error can't find the container with id 969e830b9035329126e78c6334a06a4dd154211e8015a67d3fd5fd376085817a
Sep 17 21:36:40.670776 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:40.670743    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" path="/var/lib/kubelet/pods/a903191b-4ed6-4a42-817d-39ffd25c5791/volumes"
Sep 17 21:36:40.693668 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:40.693635    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerStarted","Data":"3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec"}
Sep 17 21:36:40.693827 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:40.693702    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerStarted","Data":"969e830b9035329126e78c6334a06a4dd154211e8015a67d3fd5fd376085817a"}
Sep 17 21:36:44.405240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:44.405201    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 21:36:44.405702 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:44.405372    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:36:44.709340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:44.709269    2586 generic.go:356] "Generic (PLEG): container finished" podID="27fd0782-562c-435d-805c-067839ac18e7" containerID="3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec" exitCode=0
Sep 17 21:36:44.709505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:44.709347    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerDied","Data":"3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec"}
Sep 17 21:36:49.405877 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:49.405818    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 21:36:49.410368 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:49.410331    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 21:36:54.405861 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:54.405812    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 21:36:56.767092 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:56.767058    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerStarted","Data":"71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7"}
Sep 17 21:36:56.767505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:56.767118    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerStarted","Data":"0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f"}
Sep 17 21:36:56.767505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:56.767420    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:56.767505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:56.767468    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:36:56.768745 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:56.768720    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 21:36:56.788941 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:56.788897    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podStartSLOduration=1.5239616150000002 podStartE2EDuration="17.788883194s" totalImagesPullingTime="11.844905623s" totalInitContainerRuntime="4.420015956s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:36:56.787064656 +0000 UTC m=+2832.321004502" watchObservedRunningTime="2026-09-17 21:36:56.788883194 +0000 UTC m=+2832.322823069"
Sep 17 21:36:57.770669 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:57.770632    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 21:36:59.406069 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:59.406024    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 21:36:59.410431 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:59.410405    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 21:36:59.410518 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:36:59.410507    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:37:02.775311 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:02.775283    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:37:02.775904 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:02.775876    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 21:37:04.405258 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.405225    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 21:37:04.510585 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.510556    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:37:04.601608 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.601578    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsgzm\" (UniqueName: \"kubernetes.io/projected/357220d8-850d-4583-830f-a17aafcab5a7-kube-api-access-xsgzm\") pod \"357220d8-850d-4583-830f-a17aafcab5a7\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") "
Sep 17 21:37:04.601782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.601629    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357220d8-850d-4583-830f-a17aafcab5a7-kserve-provision-location\") pod \"357220d8-850d-4583-830f-a17aafcab5a7\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") "
Sep 17 21:37:04.601782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.601669    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-cabundle-cert\") pod \"357220d8-850d-4583-830f-a17aafcab5a7\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") "
Sep 17 21:37:04.601782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.601717    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-kube-rbac-proxy-sar-config\") pod \"357220d8-850d-4583-830f-a17aafcab5a7\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") "
Sep 17 21:37:04.601782 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.601752    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357220d8-850d-4583-830f-a17aafcab5a7-proxy-tls\") pod \"357220d8-850d-4583-830f-a17aafcab5a7\" (UID: \"357220d8-850d-4583-830f-a17aafcab5a7\") "
Sep 17 21:37:04.602039 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.602010    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357220d8-850d-4583-830f-a17aafcab5a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "357220d8-850d-4583-830f-a17aafcab5a7" (UID: "357220d8-850d-4583-830f-a17aafcab5a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:37:04.602081 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.602023    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "357220d8-850d-4583-830f-a17aafcab5a7" (UID: "357220d8-850d-4583-830f-a17aafcab5a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:37:04.602153 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.602125    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "357220d8-850d-4583-830f-a17aafcab5a7" (UID: "357220d8-850d-4583-830f-a17aafcab5a7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:37:04.603646 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.603622    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357220d8-850d-4583-830f-a17aafcab5a7-kube-api-access-xsgzm" (OuterVolumeSpecName: "kube-api-access-xsgzm") pod "357220d8-850d-4583-830f-a17aafcab5a7" (UID: "357220d8-850d-4583-830f-a17aafcab5a7"). InnerVolumeSpecName "kube-api-access-xsgzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:37:04.603853 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.603826    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357220d8-850d-4583-830f-a17aafcab5a7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "357220d8-850d-4583-830f-a17aafcab5a7" (UID: "357220d8-850d-4583-830f-a17aafcab5a7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:37:04.702404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.702377    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357220d8-850d-4583-830f-a17aafcab5a7-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:04.702404 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.702401    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:04.702625 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.702416    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/357220d8-850d-4583-830f-a17aafcab5a7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:04.702625 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.702428    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/357220d8-850d-4583-830f-a17aafcab5a7-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:04.702625 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.702442    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsgzm\" (UniqueName: \"kubernetes.io/projected/357220d8-850d-4583-830f-a17aafcab5a7-kube-api-access-xsgzm\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:04.797097 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.797069    2586 generic.go:356] "Generic (PLEG): container finished" podID="357220d8-850d-4583-830f-a17aafcab5a7" containerID="d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a" exitCode=137
Sep 17 21:37:04.797256 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.797123    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"
Sep 17 21:37:04.797256 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.797126    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerDied","Data":"d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a"}
Sep 17 21:37:04.797256 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.797246    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn" event={"ID":"357220d8-850d-4583-830f-a17aafcab5a7","Type":"ContainerDied","Data":"028c10a2c60a434b233382db55f551189416632891d9bb7d8e0613ee69757d88"}
Sep 17 21:37:04.797388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.797279    2586 scope.go:117] "RemoveContainer" containerID="074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0"
Sep 17 21:37:04.805039 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.805021    2586 scope.go:117] "RemoveContainer" containerID="d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a"
Sep 17 21:37:04.811700 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.811670    2586 scope.go:117] "RemoveContainer" containerID="0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325"
Sep 17 21:37:04.816104 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.816070    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"]
Sep 17 21:37:04.819184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.819165    2586 scope.go:117] "RemoveContainer" containerID="a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2"
Sep 17 21:37:04.820651 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.820632    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-6bdd977fb-bg7tn"]
Sep 17 21:37:04.827257 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.827242    2586 scope.go:117] "RemoveContainer" containerID="074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0"
Sep 17 21:37:04.827540 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:04.827521    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0\": container with ID starting with 074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0 not found: ID does not exist" containerID="074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0"
Sep 17 21:37:04.827606 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.827548    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0"} err="failed to get container status \"074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0\": rpc error: code = NotFound desc = could not find container \"074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0\": container with ID starting with 074cce1e9040adbcfced3d503a8d22d6690f15c610e5fe404b1e0f5f8c6a1bc0 not found: ID does not exist"
Sep 17 21:37:04.827606 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.827564    2586 scope.go:117] "RemoveContainer" containerID="d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a"
Sep 17 21:37:04.827875 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:04.827859    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a\": container with ID starting with d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a not found: ID does not exist" containerID="d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a"
Sep 17 21:37:04.827919 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.827879    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a"} err="failed to get container status \"d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a\": rpc error: code = NotFound desc = could not find container \"d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a\": container with ID starting with d92585baf8086f1141cd9c0f1710307d4e90e55518b8796bb6862f6ba6db5f1a not found: ID does not exist"
Sep 17 21:37:04.827919 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.827892    2586 scope.go:117] "RemoveContainer" containerID="0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325"
Sep 17 21:37:04.828153 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:04.828136    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325\": container with ID starting with 0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325 not found: ID does not exist" containerID="0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325"
Sep 17 21:37:04.828199 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.828160    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325"} err="failed to get container status \"0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325\": rpc error: code = NotFound desc = could not find container \"0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325\": container with ID starting with 0f8a0c16598346cf6a184da3dbb74a010c3efc610370ce9c79a8434f6377e325 not found: ID does not exist"
Sep 17 21:37:04.828199 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.828175    2586 scope.go:117] "RemoveContainer" containerID="a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2"
Sep 17 21:37:04.828844 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:04.828831    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2\": container with ID starting with a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2 not found: ID does not exist" containerID="a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2"
Sep 17 21:37:04.828896 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:04.828846    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2"} err="failed to get container status \"a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2\": rpc error: code = NotFound desc = could not find container \"a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2\": container with ID starting with a3804601829bc76303567c280e35d534a86cfc3d04ee11648a53f5b71f3126a2 not found: ID does not exist"
Sep 17 21:37:06.674718 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:06.674678    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357220d8-850d-4583-830f-a17aafcab5a7" path="/var/lib/kubelet/pods/357220d8-850d-4583-830f-a17aafcab5a7/volumes"
Sep 17 21:37:12.776152 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:12.776114    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 21:37:22.776691 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:22.776641    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 21:37:32.776006 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:32.775921    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 21:37:42.776513 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:42.776480    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:37:50.924877 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:50.924837    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"]
Sep 17 21:37:50.925279 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:50.925115    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" containerID="cri-o://0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f" gracePeriod=30
Sep 17 21:37:50.925279 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:50.925161    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kube-rbac-proxy" containerID="cri-o://71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7" gracePeriod=30
Sep 17 21:37:51.973631 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:51.973599    2586 generic.go:356] "Generic (PLEG): container finished" podID="27fd0782-562c-435d-805c-067839ac18e7" containerID="71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7" exitCode=2
Sep 17 21:37:51.974003 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:51.973657    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerDied","Data":"71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7"}
Sep 17 21:37:52.771400 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:52.771341    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Sep 17 21:37:52.776641 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:52.776614    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 17 21:37:53.664425 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.664403    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:37:53.744807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.744746    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fd0782-562c-435d-805c-067839ac18e7-kserve-provision-location\") pod \"27fd0782-562c-435d-805c-067839ac18e7\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") "
Sep 17 21:37:53.744807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.744796    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6wkr\" (UniqueName: \"kubernetes.io/projected/27fd0782-562c-435d-805c-067839ac18e7-kube-api-access-v6wkr\") pod \"27fd0782-562c-435d-805c-067839ac18e7\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") "
Sep 17 21:37:53.744990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.744820    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-cabundle-cert\") pod \"27fd0782-562c-435d-805c-067839ac18e7\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") "
Sep 17 21:37:53.744990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.744889    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-kube-rbac-proxy-sar-config\") pod \"27fd0782-562c-435d-805c-067839ac18e7\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") "
Sep 17 21:37:53.744990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.744915    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fd0782-562c-435d-805c-067839ac18e7-proxy-tls\") pod \"27fd0782-562c-435d-805c-067839ac18e7\" (UID: \"27fd0782-562c-435d-805c-067839ac18e7\") "
Sep 17 21:37:53.745256 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.745225    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27fd0782-562c-435d-805c-067839ac18e7" (UID: "27fd0782-562c-435d-805c-067839ac18e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:37:53.745321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.745250    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27fd0782-562c-435d-805c-067839ac18e7" (UID: "27fd0782-562c-435d-805c-067839ac18e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:37:53.746923 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.746897    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fd0782-562c-435d-805c-067839ac18e7-kube-api-access-v6wkr" (OuterVolumeSpecName: "kube-api-access-v6wkr") pod "27fd0782-562c-435d-805c-067839ac18e7" (UID: "27fd0782-562c-435d-805c-067839ac18e7"). InnerVolumeSpecName "kube-api-access-v6wkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:37:53.747031 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.746931    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27fd0782-562c-435d-805c-067839ac18e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27fd0782-562c-435d-805c-067839ac18e7" (UID: "27fd0782-562c-435d-805c-067839ac18e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:37:53.754918 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.754891    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fd0782-562c-435d-805c-067839ac18e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27fd0782-562c-435d-805c-067839ac18e7" (UID: "27fd0782-562c-435d-805c-067839ac18e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:37:53.845597 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.845565    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wkr\" (UniqueName: \"kubernetes.io/projected/27fd0782-562c-435d-805c-067839ac18e7-kube-api-access-v6wkr\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:53.845714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.845600    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:53.845714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.845617    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fd0782-562c-435d-805c-067839ac18e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:53.845714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.845630    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fd0782-562c-435d-805c-067839ac18e7-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:53.845714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.845644    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fd0782-562c-435d-805c-067839ac18e7-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:37:53.987092 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.987061    2586 generic.go:356] "Generic (PLEG): container finished" podID="27fd0782-562c-435d-805c-067839ac18e7" containerID="0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f" exitCode=0
Sep 17 21:37:53.987240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.987100    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerDied","Data":"0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f"}
Sep 17 21:37:53.987240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.987131    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"
Sep 17 21:37:53.987240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.987176    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk" event={"ID":"27fd0782-562c-435d-805c-067839ac18e7","Type":"ContainerDied","Data":"969e830b9035329126e78c6334a06a4dd154211e8015a67d3fd5fd376085817a"}
Sep 17 21:37:53.987240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.987202    2586 scope.go:117] "RemoveContainer" containerID="71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7"
Sep 17 21:37:53.995707 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:53.995561    2586 scope.go:117] "RemoveContainer" containerID="0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f"
Sep 17 21:37:54.002280 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.002262    2586 scope.go:117] "RemoveContainer" containerID="3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec"
Sep 17 21:37:54.009296 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.009280    2586 scope.go:117] "RemoveContainer" containerID="71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7"
Sep 17 21:37:54.009628 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:54.009610    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7\": container with ID starting with 71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7 not found: ID does not exist" containerID="71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7"
Sep 17 21:37:54.009684 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.009635    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7"} err="failed to get container status \"71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7\": rpc error: code = NotFound desc = could not find container \"71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7\": container with ID starting with 71ebe7daa1e5918e4c71fa0ded312cb3dd0bbd1da51bb29fa271f9d9adf7bec7 not found: ID does not exist"
Sep 17 21:37:54.009684 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.009650    2586 scope.go:117] "RemoveContainer" containerID="0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f"
Sep 17 21:37:54.009899 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:54.009883    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f\": container with ID starting with 0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f not found: ID does not exist" containerID="0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f"
Sep 17 21:37:54.009939 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.009905    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f"} err="failed to get container status \"0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f\": rpc error: code = NotFound desc = could not find container \"0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f\": container with ID starting with 0b387b9a4307e407ea22d74208cb01227604eb6ecb49db6d43583ea47ba97e9f not found: ID does not exist"
Sep 17 21:37:54.009939 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.009918    2586 scope.go:117] "RemoveContainer" containerID="3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec"
Sep 17 21:37:54.010151 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:54.010132    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec\": container with ID starting with 3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec not found: ID does not exist" containerID="3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec"
Sep 17 21:37:54.010188 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.010158    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec"} err="failed to get container status \"3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec\": rpc error: code = NotFound desc = could not find container \"3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec\": container with ID starting with 3e39be12d1fd98e16e755dfa64cbca19b4ebc5399ff76896694e5f8742dc72ec not found: ID does not exist"
Sep 17 21:37:54.027246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.027225    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"]
Sep 17 21:37:54.040331 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.040313    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-mdqtk"]
Sep 17 21:37:54.670788 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:54.670754    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27fd0782-562c-435d-805c-067839ac18e7" path="/var/lib/kubelet/pods/27fd0782-562c-435d-805c-067839ac18e7/volumes"
Sep 17 21:37:56.176012 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.175977    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"]
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176303    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176320    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176334    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176342    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176368    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kube-rbac-proxy"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176379    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kube-rbac-proxy"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176392    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="storage-initializer"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176402    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="storage-initializer"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176412    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-agent"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176419    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-agent"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176434    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="storage-initializer"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176441    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="storage-initializer"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176450    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176456    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176469    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kube-rbac-proxy"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176477    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kube-rbac-proxy"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176487    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="storage-initializer"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176494    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="storage-initializer"
Sep 17 21:37:56.176497 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176503    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176511    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176578    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-container"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176589    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kube-rbac-proxy"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176597    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kube-rbac-proxy"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176608    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="357220d8-850d-4583-830f-a17aafcab5a7" containerName="kserve-agent"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176619    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kube-rbac-proxy"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176627    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fd0782-562c-435d-805c-067839ac18e7" containerName="kserve-container"
Sep 17 21:37:56.177442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.176637    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a903191b-4ed6-4a42-817d-39ffd25c5791" containerName="kserve-container"
Sep 17 21:37:56.181270 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.181250    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.184835 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.184817    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:37:56.185396 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.185378    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 21:37:56.185485 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.185437    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:37:56.185552 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.185500    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:37:56.185725 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.185708    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:37:56.186218 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.186200    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:37:56.186321 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.186229    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:37:56.193888 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.193870    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"]
Sep 17 21:37:56.263449 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.263423    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.263550 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.263455    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.263550 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.263483    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx7kf\" (UniqueName: \"kubernetes.io/projected/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-api-access-gx7kf\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.263641 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.263549    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2274be89-04d3-4a27-b2b9-17f4084ca55e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.263641 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.263579    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.364686 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.364663    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.364804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.364690    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.364804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.364713    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx7kf\" (UniqueName: \"kubernetes.io/projected/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-api-access-gx7kf\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.364804 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:56.364795    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 21:37:56.364970 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.364817    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2274be89-04d3-4a27-b2b9-17f4084ca55e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.364970 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:37:56.364862    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls podName:2274be89-04d3-4a27-b2b9-17f4084ca55e nodeName:}" failed. No retries permitted until 2026-09-17 21:37:56.864841803 +0000 UTC m=+2892.398781590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-ckc6f" (UID: "2274be89-04d3-4a27-b2b9-17f4084ca55e") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 21:37:56.364970 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.364879    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.365208 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.365190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2274be89-04d3-4a27-b2b9-17f4084ca55e-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.365442 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.365423    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.365510 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.365450    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.373331 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.373308    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx7kf\" (UniqueName: \"kubernetes.io/projected/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-api-access-gx7kf\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.868566 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.868526    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:56.870724 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:56.870699    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-ckc6f\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:57.091216 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:57.091180    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:37:57.365613 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:57.365589    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"]
Sep 17 21:37:57.367878 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:37:57.367853    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2274be89_04d3_4a27_b2b9_17f4084ca55e.slice/crio-85f6997971e157ad361d99b27786afccf3a570c69c31fe6ba19cb06357c70421 WatchSource:0}: Error finding container 85f6997971e157ad361d99b27786afccf3a570c69c31fe6ba19cb06357c70421: Status 404 returned error can't find the container with id 85f6997971e157ad361d99b27786afccf3a570c69c31fe6ba19cb06357c70421
Sep 17 21:37:58.004571 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:58.004542    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerStarted","Data":"6fc99914c311e5c7a35fc2e31c585fa98bce31765a13efb10c582f15db695ac4"}
Sep 17 21:37:58.004758 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:37:58.004609    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerStarted","Data":"85f6997971e157ad361d99b27786afccf3a570c69c31fe6ba19cb06357c70421"}
Sep 17 21:38:02.022259 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:02.022223    2586 generic.go:356] "Generic (PLEG): container finished" podID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerID="6fc99914c311e5c7a35fc2e31c585fa98bce31765a13efb10c582f15db695ac4" exitCode=0
Sep 17 21:38:02.022645 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:02.022289    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerDied","Data":"6fc99914c311e5c7a35fc2e31c585fa98bce31765a13efb10c582f15db695ac4"}
Sep 17 21:38:03.028147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:03.028108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerStarted","Data":"c78c93b6b1fcc12bd1b9e2dfc67d45dbb7add1c8e1785bf3e8c051ae77c5ee41"}
Sep 17 21:38:03.028620 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:03.028177    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerStarted","Data":"a2d7354a0200868ad04b8717fa18c3592bf829845ab388ab11a1de29565d2067"}
Sep 17 21:38:03.028620 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:03.028414    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:38:03.072698 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:03.072652    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podStartSLOduration=2.610396268 podStartE2EDuration="7.072635493s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.462239225s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:38:03.071971299 +0000 UTC m=+2898.605911119" watchObservedRunningTime="2026-09-17 21:38:03.072635493 +0000 UTC m=+2898.606575302"
Sep 17 21:38:04.032286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:04.032254    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:38:04.033543 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:04.033514    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 21:38:05.035489 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:05.035450    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 21:38:10.039720 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:10.039693    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:38:10.040109 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:10.040074    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 21:38:20.040197 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:20.040152    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 21:38:28.474920 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.474880    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"]
Sep 17 21:38:28.478484 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.478459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.480916 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.480888    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 21:38:28.481020 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.481003    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 21:38:28.481312 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.481296    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 21:38:28.483876 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.483844    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.483965 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.483920    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.483965 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.483956    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94jr9\" (UniqueName: \"kubernetes.io/projected/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-api-access-94jr9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.484054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.484035    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f92cba5-0ca1-4068-bd55-cb067d217bec-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.489331 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.489311    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"]
Sep 17 21:38:28.584966 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.584938    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94jr9\" (UniqueName: \"kubernetes.io/projected/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-api-access-94jr9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.585141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.584982    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f92cba5-0ca1-4068-bd55-cb067d217bec-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.585141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.585004    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.585141 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.585035    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.585141 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:38:28.585127    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 21:38:28.585347 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:38:28.585191    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls podName:0f92cba5-0ca1-4068-bd55-cb067d217bec nodeName:}" failed. No retries permitted until 2026-09-17 21:38:29.085171969 +0000 UTC m=+2924.619111772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" (UID: "0f92cba5-0ca1-4068-bd55-cb067d217bec") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 21:38:28.585432 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.585413    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f92cba5-0ca1-4068-bd55-cb067d217bec-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.585738 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.585721    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:28.593767 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:28.593744    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94jr9\" (UniqueName: \"kubernetes.io/projected/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-api-access-94jr9\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:29.088293 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:29.088250    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:29.090636 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:29.090612    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:29.387964 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:29.387932    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:29.660953 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:29.660930    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"]
Sep 17 21:38:29.663028 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:38:29.663004    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f92cba5_0ca1_4068_bd55_cb067d217bec.slice/crio-63e99538a1f3f6b4cf1f1bb611a98b6a387f0ed6745f236618db268ac4ad40c5 WatchSource:0}: Error finding container 63e99538a1f3f6b4cf1f1bb611a98b6a387f0ed6745f236618db268ac4ad40c5: Status 404 returned error can't find the container with id 63e99538a1f3f6b4cf1f1bb611a98b6a387f0ed6745f236618db268ac4ad40c5
Sep 17 21:38:30.040517 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:30.040429    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 21:38:30.126745 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:30.126705    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" event={"ID":"0f92cba5-0ca1-4068-bd55-cb067d217bec","Type":"ContainerStarted","Data":"fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f"}
Sep 17 21:38:30.126928 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:30.126793    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" event={"ID":"0f92cba5-0ca1-4068-bd55-cb067d217bec","Type":"ContainerStarted","Data":"63e99538a1f3f6b4cf1f1bb611a98b6a387f0ed6745f236618db268ac4ad40c5"}
Sep 17 21:38:34.143150 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:34.143124    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655_0f92cba5-0ca1-4068-bd55-cb067d217bec/storage-initializer/0.log"
Sep 17 21:38:34.143529 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:34.143160    2586 generic.go:356] "Generic (PLEG): container finished" podID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerID="fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f" exitCode=1
Sep 17 21:38:34.143529 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:34.143220    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" event={"ID":"0f92cba5-0ca1-4068-bd55-cb067d217bec","Type":"ContainerDied","Data":"fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f"}
Sep 17 21:38:35.148937 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:35.148910    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655_0f92cba5-0ca1-4068-bd55-cb067d217bec/storage-initializer/0.log"
Sep 17 21:38:35.149308 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:35.148959    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" event={"ID":"0f92cba5-0ca1-4068-bd55-cb067d217bec","Type":"ContainerStarted","Data":"4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459"}
Sep 17 21:38:38.446512 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:38.446480    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"]
Sep 17 21:38:38.446984 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:38.446670    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerName="storage-initializer" containerID="cri-o://4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459" gracePeriod=30
Sep 17 21:38:38.988668 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:38.988646    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655_0f92cba5-0ca1-4068-bd55-cb067d217bec/storage-initializer/1.log"
Sep 17 21:38:38.989040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:38.989026    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655_0f92cba5-0ca1-4068-bd55-cb067d217bec/storage-initializer/0.log"
Sep 17 21:38:38.991415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:38.991400    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:39.058483 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.058411    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94jr9\" (UniqueName: \"kubernetes.io/projected/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-api-access-94jr9\") pod \"0f92cba5-0ca1-4068-bd55-cb067d217bec\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") "
Sep 17 21:38:39.058483 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.058448    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f92cba5-0ca1-4068-bd55-cb067d217bec-kserve-provision-location\") pod \"0f92cba5-0ca1-4068-bd55-cb067d217bec\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") "
Sep 17 21:38:39.058657 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.058482    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls\") pod \"0f92cba5-0ca1-4068-bd55-cb067d217bec\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") "
Sep 17 21:38:39.058657 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.058531    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-rbac-proxy-sar-config\") pod \"0f92cba5-0ca1-4068-bd55-cb067d217bec\" (UID: \"0f92cba5-0ca1-4068-bd55-cb067d217bec\") "
Sep 17 21:38:39.058742 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.058714    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f92cba5-0ca1-4068-bd55-cb067d217bec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f92cba5-0ca1-4068-bd55-cb067d217bec" (UID: "0f92cba5-0ca1-4068-bd55-cb067d217bec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:38:39.058913 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.058885    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f92cba5-0ca1-4068-bd55-cb067d217bec" (UID: "0f92cba5-0ca1-4068-bd55-cb067d217bec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:38:39.060590 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.060568    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f92cba5-0ca1-4068-bd55-cb067d217bec" (UID: "0f92cba5-0ca1-4068-bd55-cb067d217bec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:38:39.060655 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.060640    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-api-access-94jr9" (OuterVolumeSpecName: "kube-api-access-94jr9") pod "0f92cba5-0ca1-4068-bd55-cb067d217bec" (UID: "0f92cba5-0ca1-4068-bd55-cb067d217bec"). InnerVolumeSpecName "kube-api-access-94jr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:38:39.159575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.159551    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f92cba5-0ca1-4068-bd55-cb067d217bec-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:38:39.159575 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.159573    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:38:39.159749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.159587    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94jr9\" (UniqueName: \"kubernetes.io/projected/0f92cba5-0ca1-4068-bd55-cb067d217bec-kube-api-access-94jr9\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:38:39.159749 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.159600    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f92cba5-0ca1-4068-bd55-cb067d217bec-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:38:39.163780 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.163763    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655_0f92cba5-0ca1-4068-bd55-cb067d217bec/storage-initializer/1.log"
Sep 17 21:38:39.164127 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.164113    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655_0f92cba5-0ca1-4068-bd55-cb067d217bec/storage-initializer/0.log"
Sep 17 21:38:39.164192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.164144    2586 generic.go:356] "Generic (PLEG): container finished" podID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerID="4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459" exitCode=1
Sep 17 21:38:39.164192 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.164181    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" event={"ID":"0f92cba5-0ca1-4068-bd55-cb067d217bec","Type":"ContainerDied","Data":"4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459"}
Sep 17 21:38:39.164261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.164200    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"
Sep 17 21:38:39.164261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.164223    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655" event={"ID":"0f92cba5-0ca1-4068-bd55-cb067d217bec","Type":"ContainerDied","Data":"63e99538a1f3f6b4cf1f1bb611a98b6a387f0ed6745f236618db268ac4ad40c5"}
Sep 17 21:38:39.164261 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.164247    2586 scope.go:117] "RemoveContainer" containerID="4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459"
Sep 17 21:38:39.172059 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.172041    2586 scope.go:117] "RemoveContainer" containerID="fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f"
Sep 17 21:38:39.179417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.179394    2586 scope.go:117] "RemoveContainer" containerID="4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459"
Sep 17 21:38:39.179673 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:38:39.179656    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459\": container with ID starting with 4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459 not found: ID does not exist" containerID="4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459"
Sep 17 21:38:39.179735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.179680    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459"} err="failed to get container status \"4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459\": rpc error: code = NotFound desc = could not find container \"4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459\": container with ID starting with 4529e467046cec61d5f8b6af9420d54f0e8841b4a59618b22cf3efedae26b459 not found: ID does not exist"
Sep 17 21:38:39.179735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.179696    2586 scope.go:117] "RemoveContainer" containerID="fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f"
Sep 17 21:38:39.179927 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:38:39.179911    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f\": container with ID starting with fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f not found: ID does not exist" containerID="fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f"
Sep 17 21:38:39.179968 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.179930    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f"} err="failed to get container status \"fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f\": rpc error: code = NotFound desc = could not find container \"fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f\": container with ID starting with fd70f9e871259675ef46548d3e20fd9b6d13ba63f05deed06af3d2585c28992f not found: ID does not exist"
Sep 17 21:38:39.199086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.199063    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"]
Sep 17 21:38:39.202997 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:39.202976    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-8475db5d97-ns655"]
Sep 17 21:38:40.040606 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:40.040567    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 21:38:40.670485 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:40.670452    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" path="/var/lib/kubelet/pods/0f92cba5-0ca1-4068-bd55-cb067d217bec/volumes"
Sep 17 21:38:50.041004 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:50.040969    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:38:57.535018 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:57.534933    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"]
Sep 17 21:38:57.535565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:57.535229    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" containerID="cri-o://a2d7354a0200868ad04b8717fa18c3592bf829845ab388ab11a1de29565d2067" gracePeriod=30
Sep 17 21:38:57.535565 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:57.535257    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kube-rbac-proxy" containerID="cri-o://c78c93b6b1fcc12bd1b9e2dfc67d45dbb7add1c8e1785bf3e8c051ae77c5ee41" gracePeriod=30
Sep 17 21:38:58.238735 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:58.238699    2586 generic.go:356] "Generic (PLEG): container finished" podID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerID="c78c93b6b1fcc12bd1b9e2dfc67d45dbb7add1c8e1785bf3e8c051ae77c5ee41" exitCode=2
Sep 17 21:38:58.238906 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:38:58.238766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerDied","Data":"c78c93b6b1fcc12bd1b9e2dfc67d45dbb7add1c8e1785bf3e8c051ae77c5ee41"}
Sep 17 21:39:00.036629 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.036586    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.62:8643/healthz\": dial tcp 10.132.0.62:8643: connect: connection refused"
Sep 17 21:39:00.040953 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.040923    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.62:8080: connect: connection refused"
Sep 17 21:39:00.248130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.248096    2586 generic.go:356] "Generic (PLEG): container finished" podID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerID="a2d7354a0200868ad04b8717fa18c3592bf829845ab388ab11a1de29565d2067" exitCode=0
Sep 17 21:39:00.248269 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.248162    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerDied","Data":"a2d7354a0200868ad04b8717fa18c3592bf829845ab388ab11a1de29565d2067"}
Sep 17 21:39:00.277279 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.277260    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:39:00.411184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.411159    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-rbac-proxy-sar-config\") pod \"2274be89-04d3-4a27-b2b9-17f4084ca55e\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") "
Sep 17 21:39:00.411384 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.411197    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx7kf\" (UniqueName: \"kubernetes.io/projected/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-api-access-gx7kf\") pod \"2274be89-04d3-4a27-b2b9-17f4084ca55e\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") "
Sep 17 21:39:00.411384 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.411216    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls\") pod \"2274be89-04d3-4a27-b2b9-17f4084ca55e\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") "
Sep 17 21:39:00.411384 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.411237    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-cabundle-cert\") pod \"2274be89-04d3-4a27-b2b9-17f4084ca55e\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") "
Sep 17 21:39:00.411384 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.411321    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2274be89-04d3-4a27-b2b9-17f4084ca55e-kserve-provision-location\") pod \"2274be89-04d3-4a27-b2b9-17f4084ca55e\" (UID: \"2274be89-04d3-4a27-b2b9-17f4084ca55e\") "
Sep 17 21:39:00.411648 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.411617    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2274be89-04d3-4a27-b2b9-17f4084ca55e" (UID: "2274be89-04d3-4a27-b2b9-17f4084ca55e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:39:00.411720 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.411625    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2274be89-04d3-4a27-b2b9-17f4084ca55e" (UID: "2274be89-04d3-4a27-b2b9-17f4084ca55e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:39:00.413389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.413332    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-api-access-gx7kf" (OuterVolumeSpecName: "kube-api-access-gx7kf") pod "2274be89-04d3-4a27-b2b9-17f4084ca55e" (UID: "2274be89-04d3-4a27-b2b9-17f4084ca55e"). InnerVolumeSpecName "kube-api-access-gx7kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:39:00.413389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.413373    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2274be89-04d3-4a27-b2b9-17f4084ca55e" (UID: "2274be89-04d3-4a27-b2b9-17f4084ca55e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:39:00.421516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.421491    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2274be89-04d3-4a27-b2b9-17f4084ca55e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2274be89-04d3-4a27-b2b9-17f4084ca55e" (UID: "2274be89-04d3-4a27-b2b9-17f4084ca55e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:39:00.512589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.512558    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:39:00.512589 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.512589    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx7kf\" (UniqueName: \"kubernetes.io/projected/2274be89-04d3-4a27-b2b9-17f4084ca55e-kube-api-access-gx7kf\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:39:00.512721 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.512600    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2274be89-04d3-4a27-b2b9-17f4084ca55e-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:39:00.512721 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.512610    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2274be89-04d3-4a27-b2b9-17f4084ca55e-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:39:00.512721 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:00.512617    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2274be89-04d3-4a27-b2b9-17f4084ca55e-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:39:01.253860 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:01.253829    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f" event={"ID":"2274be89-04d3-4a27-b2b9-17f4084ca55e","Type":"ContainerDied","Data":"85f6997971e157ad361d99b27786afccf3a570c69c31fe6ba19cb06357c70421"}
Sep 17 21:39:01.254277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:01.253878    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"
Sep 17 21:39:01.254277 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:01.253893    2586 scope.go:117] "RemoveContainer" containerID="c78c93b6b1fcc12bd1b9e2dfc67d45dbb7add1c8e1785bf3e8c051ae77c5ee41"
Sep 17 21:39:01.261634 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:01.261615    2586 scope.go:117] "RemoveContainer" containerID="a2d7354a0200868ad04b8717fa18c3592bf829845ab388ab11a1de29565d2067"
Sep 17 21:39:01.268509 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:01.268492    2586 scope.go:117] "RemoveContainer" containerID="6fc99914c311e5c7a35fc2e31c585fa98bce31765a13efb10c582f15db695ac4"
Sep 17 21:39:01.271415 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:01.271394    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"]
Sep 17 21:39:01.280054 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:01.280028    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-ckc6f"]
Sep 17 21:39:02.670249 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.670214    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" path="/var/lib/kubelet/pods/2274be89-04d3-4a27-b2b9-17f4084ca55e/volumes"
Sep 17 21:39:02.820086 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820053    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"]
Sep 17 21:39:02.820316 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820305    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kube-rbac-proxy"
Sep 17 21:39:02.820372 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820319    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kube-rbac-proxy"
Sep 17 21:39:02.820372 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820329    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container"
Sep 17 21:39:02.820372 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820335    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container"
Sep 17 21:39:02.820372 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820342    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="storage-initializer"
Sep 17 21:39:02.820372 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820347    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="storage-initializer"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820374    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerName="storage-initializer"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820382    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerName="storage-initializer"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820388    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerName="storage-initializer"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820393    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerName="storage-initializer"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820430    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerName="storage-initializer"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820437    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kube-rbac-proxy"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820446    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f92cba5-0ca1-4068-bd55-cb067d217bec" containerName="storage-initializer"
Sep 17 21:39:02.820534 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.820456    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2274be89-04d3-4a27-b2b9-17f4084ca55e" containerName="kserve-container"
Sep 17 21:39:02.824888 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.824868    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:02.829644 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.829616    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:39:02.829779 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.829627    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:39:02.830266 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.830246    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:39:02.830342 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.830253    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:39:02.830428 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.830380    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:39:02.830488 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.830435    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 21:39:02.830629 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.830612    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 21:39:02.837288 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.837267    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"]
Sep 17 21:39:02.929182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.929124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:02.929182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.929155    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b0ab8be-3597-4335-b331-0cf9d38cc095-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:02.929182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.929177    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5gj\" (UniqueName: \"kubernetes.io/projected/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-api-access-9j5gj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:02.929439 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.929193    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:02.929439 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:02.929319    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.030390 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.030346    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.030537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.030409    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.030537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.030440    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b0ab8be-3597-4335-b331-0cf9d38cc095-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.030537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.030468    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5gj\" (UniqueName: \"kubernetes.io/projected/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-api-access-9j5gj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.030537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.030494    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.030749 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:39:03.030618    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 21:39:03.030749 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:39:03.030683    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls podName:0b0ab8be-3597-4335-b331-0cf9d38cc095 nodeName:}" failed. No retries permitted until 2026-09-17 21:39:03.530663037 +0000 UTC m=+2959.064602827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" (UID: "0b0ab8be-3597-4335-b331-0cf9d38cc095") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 17 21:39:03.030852 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.030822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b0ab8be-3597-4335-b331-0cf9d38cc095-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.031037 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.031018    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.031076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.031034    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.038937 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.038903    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5gj\" (UniqueName: \"kubernetes.io/projected/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-api-access-9j5gj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.533725 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.533695    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.536013 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.535985    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:03.734711 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:03.734680    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:04.016155 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:04.016124    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"]
Sep 17 21:39:04.018860 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:39:04.018833    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0ab8be_3597_4335_b331_0cf9d38cc095.slice/crio-e9e121898371109ad4f2b055ad3a19e269a35067876fc55f2c4da54ce1a34e37 WatchSource:0}: Error finding container e9e121898371109ad4f2b055ad3a19e269a35067876fc55f2c4da54ce1a34e37: Status 404 returned error can't find the container with id e9e121898371109ad4f2b055ad3a19e269a35067876fc55f2c4da54ce1a34e37
Sep 17 21:39:04.266706 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:04.266671    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerStarted","Data":"5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee"}
Sep 17 21:39:04.266847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:04.266735    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerStarted","Data":"e9e121898371109ad4f2b055ad3a19e269a35067876fc55f2c4da54ce1a34e37"}
Sep 17 21:39:09.285471 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:09.285433    2586 generic.go:356] "Generic (PLEG): container finished" podID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerID="5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee" exitCode=0
Sep 17 21:39:09.285860 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:09.285508    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerDied","Data":"5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee"}
Sep 17 21:39:10.291549 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:10.291510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerStarted","Data":"36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e"}
Sep 17 21:39:10.291940 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:10.291577    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerStarted","Data":"ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa"}
Sep 17 21:39:10.291940 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:10.291776    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:10.312136 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:10.312093    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podStartSLOduration=3.810630268 podStartE2EDuration="8.312078347s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.501448079s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:39:10.310308142 +0000 UTC m=+2965.844247964" watchObservedRunningTime="2026-09-17 21:39:10.312078347 +0000 UTC m=+2965.846018156"
Sep 17 21:39:11.296165 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:11.296130    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:11.297091 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:11.297063    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 17 21:39:12.299971 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:12.299936    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 17 21:39:17.304651 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:17.304621    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:39:17.305145 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:17.305116    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 17 21:39:27.305632 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:27.305592    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 17 21:39:37.305932 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:37.305894    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 17 21:39:47.306018 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:47.305977    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 17 21:39:57.306539 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:39:57.306506    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:40:04.364847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:04.364817    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"]
Sep 17 21:40:04.365222 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:04.365033    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container" containerID="cri-o://ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa" gracePeriod=30
Sep 17 21:40:04.365222 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:04.365080    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kube-rbac-proxy" containerID="cri-o://36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e" gracePeriod=30
Sep 17 21:40:04.489725 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:04.489696    2586 generic.go:356] "Generic (PLEG): container finished" podID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerID="36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e" exitCode=2
Sep 17 21:40:04.489826 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:04.489768    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerDied","Data":"36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e"}
Sep 17 21:40:07.099183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.099158    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:40:07.179562 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.179506    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j5gj\" (UniqueName: \"kubernetes.io/projected/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-api-access-9j5gj\") pod \"0b0ab8be-3597-4335-b331-0cf9d38cc095\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") "
Sep 17 21:40:07.179562 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.179535    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-rbac-proxy-sar-config\") pod \"0b0ab8be-3597-4335-b331-0cf9d38cc095\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") "
Sep 17 21:40:07.179714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.179569    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-cabundle-cert\") pod \"0b0ab8be-3597-4335-b331-0cf9d38cc095\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") "
Sep 17 21:40:07.179714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.179603    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b0ab8be-3597-4335-b331-0cf9d38cc095-kserve-provision-location\") pod \"0b0ab8be-3597-4335-b331-0cf9d38cc095\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") "
Sep 17 21:40:07.179714 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.179630    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls\") pod \"0b0ab8be-3597-4335-b331-0cf9d38cc095\" (UID: \"0b0ab8be-3597-4335-b331-0cf9d38cc095\") "
Sep 17 21:40:07.179941 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.179911    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b0ab8be-3597-4335-b331-0cf9d38cc095" (UID: "0b0ab8be-3597-4335-b331-0cf9d38cc095"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:40:07.180047 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.179957    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b0ab8be-3597-4335-b331-0cf9d38cc095" (UID: "0b0ab8be-3597-4335-b331-0cf9d38cc095"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:40:07.181719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.181691    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-api-access-9j5gj" (OuterVolumeSpecName: "kube-api-access-9j5gj") pod "0b0ab8be-3597-4335-b331-0cf9d38cc095" (UID: "0b0ab8be-3597-4335-b331-0cf9d38cc095"). InnerVolumeSpecName "kube-api-access-9j5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:40:07.181719 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.181691    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b0ab8be-3597-4335-b331-0cf9d38cc095" (UID: "0b0ab8be-3597-4335-b331-0cf9d38cc095"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:40:07.190186 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.190159    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0ab8be-3597-4335-b331-0cf9d38cc095-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b0ab8be-3597-4335-b331-0cf9d38cc095" (UID: "0b0ab8be-3597-4335-b331-0cf9d38cc095"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:40:07.280699 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.280675    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5gj\" (UniqueName: \"kubernetes.io/projected/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-api-access-9j5gj\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:07.280699 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.280695    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:07.280819 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.280705    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b0ab8be-3597-4335-b331-0cf9d38cc095-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:07.280819 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.280714    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b0ab8be-3597-4335-b331-0cf9d38cc095-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:07.280819 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.280723    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b0ab8be-3597-4335-b331-0cf9d38cc095-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:07.506573 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.506482    2586 generic.go:356] "Generic (PLEG): container finished" podID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerID="ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa" exitCode=0
Sep 17 21:40:07.506573 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.506534    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"
Sep 17 21:40:07.506759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.506569    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerDied","Data":"ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa"}
Sep 17 21:40:07.506759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.506622    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz" event={"ID":"0b0ab8be-3597-4335-b331-0cf9d38cc095","Type":"ContainerDied","Data":"e9e121898371109ad4f2b055ad3a19e269a35067876fc55f2c4da54ce1a34e37"}
Sep 17 21:40:07.506759 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.506647    2586 scope.go:117] "RemoveContainer" containerID="36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e"
Sep 17 21:40:07.514710 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.514692    2586 scope.go:117] "RemoveContainer" containerID="ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa"
Sep 17 21:40:07.521830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.521814    2586 scope.go:117] "RemoveContainer" containerID="5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee"
Sep 17 21:40:07.527438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.527407    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"]
Sep 17 21:40:07.528797 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.528782    2586 scope.go:117] "RemoveContainer" containerID="36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e"
Sep 17 21:40:07.529019 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:07.529003    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e\": container with ID starting with 36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e not found: ID does not exist" containerID="36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e"
Sep 17 21:40:07.529061 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.529027    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e"} err="failed to get container status \"36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e\": rpc error: code = NotFound desc = could not find container \"36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e\": container with ID starting with 36670c57d98d4313eb8227f1e3d9d560628344171d4a5929ebe9c9c62847ce5e not found: ID does not exist"
Sep 17 21:40:07.529061 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.529044    2586 scope.go:117] "RemoveContainer" containerID="ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa"
Sep 17 21:40:07.529270 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:07.529254    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa\": container with ID starting with ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa not found: ID does not exist" containerID="ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa"
Sep 17 21:40:07.529306 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.529274    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa"} err="failed to get container status \"ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa\": rpc error: code = NotFound desc = could not find container \"ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa\": container with ID starting with ecc8c9317371ca3b49c9b6fb15e496f94cf3fc478d55b8bc443d31ca5a78caaa not found: ID does not exist"
Sep 17 21:40:07.529306 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.529286    2586 scope.go:117] "RemoveContainer" containerID="5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee"
Sep 17 21:40:07.529554 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:07.529531    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee\": container with ID starting with 5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee not found: ID does not exist" containerID="5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee"
Sep 17 21:40:07.529613 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.529565    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee"} err="failed to get container status \"5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee\": rpc error: code = NotFound desc = could not find container \"5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee\": container with ID starting with 5a5b43604e26adff5e030c6ffedcd623541e999c45b638f02fc9d05d6f5297ee not found: ID does not exist"
Sep 17 21:40:07.531146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:07.531125    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-t7nlz"]
Sep 17 21:40:08.670786 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:08.670758    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" path="/var/lib/kubelet/pods/0b0ab8be-3597-4335-b331-0cf9d38cc095/volumes"
Sep 17 21:40:09.575017 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.574982    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"]
Sep 17 21:40:09.575240 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575228    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kube-rbac-proxy"
Sep 17 21:40:09.575290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575241    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kube-rbac-proxy"
Sep 17 21:40:09.575290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575252    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="storage-initializer"
Sep 17 21:40:09.575290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575259    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="storage-initializer"
Sep 17 21:40:09.575290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575274    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container"
Sep 17 21:40:09.575290 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575279    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container"
Sep 17 21:40:09.575481 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575317    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kube-rbac-proxy"
Sep 17 21:40:09.575481 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.575328    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0ab8be-3597-4335-b331-0cf9d38cc095" containerName="kserve-container"
Sep 17 21:40:09.579574 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.579558    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.581783 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.581762    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 21:40:09.582236 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.582219    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 21:40:09.582297 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.582235    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 21:40:09.582297 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.582271    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lmw5m\""
Sep 17 21:40:09.582830 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.582807    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 21:40:09.582940 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.582853    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 21:40:09.582996 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.582949    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 21:40:09.587726 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.587705    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"]
Sep 17 21:40:09.595644 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.595617    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.595739 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.595709    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.595792 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.595749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvjf\" (UniqueName: \"kubernetes.io/projected/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-api-access-svvjf\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.595792 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.595785    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.595868 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.595813    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697063 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697032    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697063 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697072    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvjf\" (UniqueName: \"kubernetes.io/projected/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-api-access-svvjf\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697522 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697522 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697522 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697147    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697522 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:09.697206    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 21:40:09.697522 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:09.697279    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls podName:65fd0171-6ab8-4cb2-9801-d3444235ba2e nodeName:}" failed. No retries permitted until 2026-09-17 21:40:10.197258328 +0000 UTC m=+3025.731198119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls") pod "isvc-pmml-predictor-8674795f59-nqbzg" (UID: "65fd0171-6ab8-4cb2-9801-d3444235ba2e") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 21:40:09.697694 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697891 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697869    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.697928 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.697869    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:09.706095 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:09.706073    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvjf\" (UniqueName: \"kubernetes.io/projected/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-api-access-svvjf\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:10.200160 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.200130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:10.202438 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.202420    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-nqbzg\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:10.489982 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.489898    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:10.680831 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.680799    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"]
Sep 17 21:40:10.684276 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.684257    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.686448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.686429    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 21:40:10.686558 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.686469    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 21:40:10.686717 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.686703    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 21:40:10.696907 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.696887    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"]
Sep 17 21:40:10.704243 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.704220    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.704552 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.704278    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2253c225-a592-4444-8708-223cae58e754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.704552 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.704338    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2253c225-a592-4444-8708-223cae58e754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.704552 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.704381    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgh5\" (UniqueName: \"kubernetes.io/projected/2253c225-a592-4444-8708-223cae58e754-kube-api-access-9cgh5\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.760333 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.760308    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"]
Sep 17 21:40:10.762672 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:40:10.762649    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65fd0171_6ab8_4cb2_9801_d3444235ba2e.slice/crio-3ce26ca5598d90521638e60837b34b5046998367c89be687334c656703803ed2 WatchSource:0}: Error finding container 3ce26ca5598d90521638e60837b34b5046998367c89be687334c656703803ed2: Status 404 returned error can't find the container with id 3ce26ca5598d90521638e60837b34b5046998367c89be687334c656703803ed2
Sep 17 21:40:10.805038 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.805014    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.805130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.805060    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2253c225-a592-4444-8708-223cae58e754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.805130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.805096    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2253c225-a592-4444-8708-223cae58e754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.805130 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.805114    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgh5\" (UniqueName: \"kubernetes.io/projected/2253c225-a592-4444-8708-223cae58e754-kube-api-access-9cgh5\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.805234 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:10.805165    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 21:40:10.805234 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:10.805216    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls podName:2253c225-a592-4444-8708-223cae58e754 nodeName:}" failed. No retries permitted until 2026-09-17 21:40:11.30520148 +0000 UTC m=+3026.839141292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" (UID: "2253c225-a592-4444-8708-223cae58e754") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 21:40:10.805603 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.805576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2253c225-a592-4444-8708-223cae58e754-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.805849 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.805827    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2253c225-a592-4444-8708-223cae58e754-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:10.815418 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:10.815396    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgh5\" (UniqueName: \"kubernetes.io/projected/2253c225-a592-4444-8708-223cae58e754-kube-api-access-9cgh5\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:11.308605 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:11.308573    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:11.310963 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:11.310933    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:11.524090 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:11.524052    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerStarted","Data":"5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4"}
Sep 17 21:40:11.524271 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:11.524122    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerStarted","Data":"3ce26ca5598d90521638e60837b34b5046998367c89be687334c656703803ed2"}
Sep 17 21:40:11.596848 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:11.596774    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:11.865897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:11.865873    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"]
Sep 17 21:40:11.867636 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:40:11.867611    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2253c225_a592_4444_8708_223cae58e754.slice/crio-a9e61558739c832436f2b7c0b49768717e2920fd18dea7069643ff980a20bed0 WatchSource:0}: Error finding container a9e61558739c832436f2b7c0b49768717e2920fd18dea7069643ff980a20bed0: Status 404 returned error can't find the container with id a9e61558739c832436f2b7c0b49768717e2920fd18dea7069643ff980a20bed0
Sep 17 21:40:12.529553 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:12.529518    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" event={"ID":"2253c225-a592-4444-8708-223cae58e754","Type":"ContainerStarted","Data":"07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557"}
Sep 17 21:40:12.529726 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:12.529585    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" event={"ID":"2253c225-a592-4444-8708-223cae58e754","Type":"ContainerStarted","Data":"a9e61558739c832436f2b7c0b49768717e2920fd18dea7069643ff980a20bed0"}
Sep 17 21:40:14.537814 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:14.537787    2586 generic.go:356] "Generic (PLEG): container finished" podID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerID="5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4" exitCode=0
Sep 17 21:40:14.538111 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:14.537831    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerDied","Data":"5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4"}
Sep 17 21:40:19.562745 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:19.562717    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml_2253c225-a592-4444-8708-223cae58e754/storage-initializer/0.log"
Sep 17 21:40:19.563147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:19.562759    2586 generic.go:356] "Generic (PLEG): container finished" podID="2253c225-a592-4444-8708-223cae58e754" containerID="07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557" exitCode=1
Sep 17 21:40:19.563147 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:19.562812    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" event={"ID":"2253c225-a592-4444-8708-223cae58e754","Type":"ContainerDied","Data":"07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557"}
Sep 17 21:40:20.701676 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:20.701642    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"]
Sep 17 21:40:21.851514 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.851483    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"]
Sep 17 21:40:21.854728 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.854705    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.857168 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.857151    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 21:40:21.857885 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.857857    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 21:40:21.864802 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.864780    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"]
Sep 17 21:40:21.886799 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.886772    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.886904 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.886810    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.886904 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.886830    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmcv6\" (UniqueName: \"kubernetes.io/projected/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-api-access-zmcv6\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.887035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.886917    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.887035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.886963    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.987598 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.987517    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.987598 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.987562    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.987598 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.987586    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmcv6\" (UniqueName: \"kubernetes.io/projected/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-api-access-zmcv6\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.987878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.987613    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.987878 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.987643    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.987878 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:21.987682    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 17 21:40:21.987878 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:21.987748    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls podName:70b97cae-d5dc-41b9-9b4d-add78d57c31d nodeName:}" failed. No retries permitted until 2026-09-17 21:40:22.48772889 +0000 UTC m=+3038.021668695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-wbczm" (UID: "70b97cae-d5dc-41b9-9b4d-add78d57c31d") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 17 21:40:21.988089 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.988061    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.988292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.988269    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.988523 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.988497    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:21.996604 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:21.996585    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmcv6\" (UniqueName: \"kubernetes.io/projected/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-api-access-zmcv6\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:22.490872 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.490833    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:22.493187 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.493164    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-wbczm\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:22.575278 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.575251    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml_2253c225-a592-4444-8708-223cae58e754/storage-initializer/0.log"
Sep 17 21:40:22.575488 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.575380    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" event={"ID":"2253c225-a592-4444-8708-223cae58e754","Type":"ContainerStarted","Data":"1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742"}
Sep 17 21:40:22.575488 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.575456    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" podUID="2253c225-a592-4444-8708-223cae58e754" containerName="storage-initializer" containerID="cri-o://1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742" gracePeriod=30
Sep 17 21:40:22.577579 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.577559    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerStarted","Data":"d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a"}
Sep 17 21:40:22.577688 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.577602    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerStarted","Data":"8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e"}
Sep 17 21:40:22.577790 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.577771    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:22.620441 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.620402    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podStartSLOduration=2.7974421080000003 podStartE2EDuration="13.620387519s" totalImagesPullingTime="7.121946718s" totalInitContainerRuntime="3.700998693s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:40:22.619536375 +0000 UTC m=+3038.153476185" watchObservedRunningTime="2026-09-17 21:40:22.620387519 +0000 UTC m=+3038.154327322"
Sep 17 21:40:22.764710 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:22.764639    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:23.037011 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:23.036934    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"]
Sep 17 21:40:23.039769 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:40:23.039743    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b97cae_d5dc_41b9_9b4d_add78d57c31d.slice/crio-2190240a3501b7596f2b8dd75d1e334dadb600bb50efe67fbd33d4a0ac004e84 WatchSource:0}: Error finding container 2190240a3501b7596f2b8dd75d1e334dadb600bb50efe67fbd33d4a0ac004e84: Status 404 returned error can't find the container with id 2190240a3501b7596f2b8dd75d1e334dadb600bb50efe67fbd33d4a0ac004e84
Sep 17 21:40:23.583189 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:23.583150    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerStarted","Data":"5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98"}
Sep 17 21:40:23.583417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:23.583211    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerStarted","Data":"2190240a3501b7596f2b8dd75d1e334dadb600bb50efe67fbd33d4a0ac004e84"}
Sep 17 21:40:23.583417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:23.583263    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:23.584771 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:23.584743    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:40:24.586815 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:24.586776    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:40:26.407736 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.407715    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml_2253c225-a592-4444-8708-223cae58e754/storage-initializer/1.log"
Sep 17 21:40:26.408120 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.408102    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml_2253c225-a592-4444-8708-223cae58e754/storage-initializer/0.log"
Sep 17 21:40:26.410408 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.410390    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:26.519142 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519114    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls\") pod \"2253c225-a592-4444-8708-223cae58e754\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") "
Sep 17 21:40:26.519334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519168    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2253c225-a592-4444-8708-223cae58e754-kserve-provision-location\") pod \"2253c225-a592-4444-8708-223cae58e754\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") "
Sep 17 21:40:26.519334 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519195    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2253c225-a592-4444-8708-223cae58e754-kube-rbac-proxy-sar-config\") pod \"2253c225-a592-4444-8708-223cae58e754\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") "
Sep 17 21:40:26.519492 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519342    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cgh5\" (UniqueName: \"kubernetes.io/projected/2253c225-a592-4444-8708-223cae58e754-kube-api-access-9cgh5\") pod \"2253c225-a592-4444-8708-223cae58e754\" (UID: \"2253c225-a592-4444-8708-223cae58e754\") "
Sep 17 21:40:26.519554 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519488    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2253c225-a592-4444-8708-223cae58e754-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2253c225-a592-4444-8708-223cae58e754" (UID: "2253c225-a592-4444-8708-223cae58e754"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:40:26.519611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519569    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2253c225-a592-4444-8708-223cae58e754-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2253c225-a592-4444-8708-223cae58e754" (UID: "2253c225-a592-4444-8708-223cae58e754"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:40:26.519686 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519671    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2253c225-a592-4444-8708-223cae58e754-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:26.519736 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.519693    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2253c225-a592-4444-8708-223cae58e754-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:26.521413 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.521339    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2253c225-a592-4444-8708-223cae58e754" (UID: "2253c225-a592-4444-8708-223cae58e754"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:40:26.521520 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.521456    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2253c225-a592-4444-8708-223cae58e754-kube-api-access-9cgh5" (OuterVolumeSpecName: "kube-api-access-9cgh5") pod "2253c225-a592-4444-8708-223cae58e754" (UID: "2253c225-a592-4444-8708-223cae58e754"). InnerVolumeSpecName "kube-api-access-9cgh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:40:26.594964 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.594942    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml_2253c225-a592-4444-8708-223cae58e754/storage-initializer/1.log"
Sep 17 21:40:26.595337 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.595315    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml_2253c225-a592-4444-8708-223cae58e754/storage-initializer/0.log"
Sep 17 21:40:26.595458 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.595350    2586 generic.go:356] "Generic (PLEG): container finished" podID="2253c225-a592-4444-8708-223cae58e754" containerID="1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742" exitCode=1
Sep 17 21:40:26.595458 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.595428    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"
Sep 17 21:40:26.595527 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.595456    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" event={"ID":"2253c225-a592-4444-8708-223cae58e754","Type":"ContainerDied","Data":"1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742"}
Sep 17 21:40:26.595567 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.595526    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml" event={"ID":"2253c225-a592-4444-8708-223cae58e754","Type":"ContainerDied","Data":"a9e61558739c832436f2b7c0b49768717e2920fd18dea7069643ff980a20bed0"}
Sep 17 21:40:26.595601 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.595566    2586 scope.go:117] "RemoveContainer" containerID="1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742"
Sep 17 21:40:26.603825 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.603807    2586 scope.go:117] "RemoveContainer" containerID="07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557"
Sep 17 21:40:26.611216 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.611195    2586 scope.go:117] "RemoveContainer" containerID="1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742"
Sep 17 21:40:26.611542 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:26.611524    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742\": container with ID starting with 1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742 not found: ID does not exist" containerID="1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742"
Sep 17 21:40:26.611610 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.611548    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742"} err="failed to get container status \"1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742\": rpc error: code = NotFound desc = could not find container \"1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742\": container with ID starting with 1648e7150b3913a3bd64134e528177b328fbe63c7db67bea477e7bf85fff9742 not found: ID does not exist"
Sep 17 21:40:26.611610 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.611563    2586 scope.go:117] "RemoveContainer" containerID="07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557"
Sep 17 21:40:26.611821 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:40:26.611807    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557\": container with ID starting with 07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557 not found: ID does not exist" containerID="07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557"
Sep 17 21:40:26.611872 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.611824    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557"} err="failed to get container status \"07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557\": rpc error: code = NotFound desc = could not find container \"07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557\": container with ID starting with 07635a9f00dec8238bf1bfcec87e2fe767fae85b58fd716394b6aebba3472557 not found: ID does not exist"
Sep 17 21:40:26.620169 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.620153    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2253c225-a592-4444-8708-223cae58e754-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:26.620246 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.620172    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgh5\" (UniqueName: \"kubernetes.io/projected/2253c225-a592-4444-8708-223cae58e754-kube-api-access-9cgh5\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:40:26.631340 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.631313    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"]
Sep 17 21:40:26.637380 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.637328    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6d7b97c496-6bmml"]
Sep 17 21:40:26.671093 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:26.671060    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2253c225-a592-4444-8708-223cae58e754" path="/var/lib/kubelet/pods/2253c225-a592-4444-8708-223cae58e754/volumes"
Sep 17 21:40:27.602221 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:27.602141    2586 generic.go:356] "Generic (PLEG): container finished" podID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerID="5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98" exitCode=0
Sep 17 21:40:27.602661 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:27.602218    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerDied","Data":"5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98"}
Sep 17 21:40:28.611413 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:28.611376    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerStarted","Data":"0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a"}
Sep 17 21:40:28.611807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:28.611465    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerStarted","Data":"06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2"}
Sep 17 21:40:28.611807 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:28.611660    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:28.632720 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:28.632678    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podStartSLOduration=4.043878714 podStartE2EDuration="7.63266327s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.588784556s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:40:28.630861384 +0000 UTC m=+3044.164801226" watchObservedRunningTime="2026-09-17 21:40:28.63266327 +0000 UTC m=+3044.166603129"
Sep 17 21:40:29.591287 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:29.591254    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:40:29.591783 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:29.591756    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:40:29.615690 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:29.615662    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:29.616610 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:29.616582    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:40:30.619920 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:30.619880    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:40:35.624035 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:35.624007    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:40:35.624506 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:35.624480    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:40:39.592517 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:39.592473    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:40:45.625264 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:45.625220    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:40:49.591883 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:49.591847    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:40:52.534700 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:52.534666    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:40:52.536515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:52.536491    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:40:55.625092 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:55.625052    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:40:59.591872 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:40:59.591833    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:41:05.624931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:05.624889    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:41:09.592487 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:09.592448    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:41:15.625108 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:15.625070    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:41:19.592457 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:19.592416    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:41:25.624622 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:25.624584    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:41:29.592374 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:29.592320    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:41:35.624505 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:35.624468    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:41:39.592286 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:39.592248    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 17 21:41:41.667047 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:41.667000    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.67:8080: connect: connection refused"
Sep 17 21:41:49.592278 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:49.592249    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:41:51.668097 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:41:51.668065    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:42:01.362784 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:01.362751    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"]
Sep 17 21:42:01.363178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:01.362996    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container" containerID="cri-o://8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e" gracePeriod=30
Sep 17 21:42:01.363178 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:01.363038    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kube-rbac-proxy" containerID="cri-o://d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a" gracePeriod=30
Sep 17 21:42:01.958889 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:01.958857    2586 generic.go:356] "Generic (PLEG): container finished" podID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerID="d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a" exitCode=2
Sep 17 21:42:01.959069 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:01.958901    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerDied","Data":"d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a"}
Sep 17 21:42:03.554826 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:03.554793    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"]
Sep 17 21:42:03.555267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:03.555154    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container" containerID="cri-o://06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2" gracePeriod=30
Sep 17 21:42:03.555267 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:03.555195    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kube-rbac-proxy" containerID="cri-o://0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a" gracePeriod=30
Sep 17 21:42:03.968220 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:03.968192    2586 generic.go:356] "Generic (PLEG): container finished" podID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerID="0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a" exitCode=2
Sep 17 21:42:03.968398 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:03.968266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerDied","Data":"0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a"}
Sep 17 21:42:04.587658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:04.587621    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Sep 17 21:42:05.198905 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.198879    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:42:05.224202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224176    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls\") pod \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") "
Sep 17 21:42:05.224397 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224217    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-cabundle-cert\") pod \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") "
Sep 17 21:42:05.224397 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224237    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-rbac-proxy-sar-config\") pod \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") "
Sep 17 21:42:05.224397 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224266    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svvjf\" (UniqueName: \"kubernetes.io/projected/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-api-access-svvjf\") pod \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") "
Sep 17 21:42:05.224397 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224317    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kserve-provision-location\") pod \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\" (UID: \"65fd0171-6ab8-4cb2-9801-d3444235ba2e\") "
Sep 17 21:42:05.224676 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224650    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65fd0171-6ab8-4cb2-9801-d3444235ba2e" (UID: "65fd0171-6ab8-4cb2-9801-d3444235ba2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:42:05.224747 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224685    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65fd0171-6ab8-4cb2-9801-d3444235ba2e" (UID: "65fd0171-6ab8-4cb2-9801-d3444235ba2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:05.224804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.224745    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65fd0171-6ab8-4cb2-9801-d3444235ba2e" (UID: "65fd0171-6ab8-4cb2-9801-d3444235ba2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:42:05.226982 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.226947    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-api-access-svvjf" (OuterVolumeSpecName: "kube-api-access-svvjf") pod "65fd0171-6ab8-4cb2-9801-d3444235ba2e" (UID: "65fd0171-6ab8-4cb2-9801-d3444235ba2e"). InnerVolumeSpecName "kube-api-access-svvjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:42:05.227500 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.227475    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65fd0171-6ab8-4cb2-9801-d3444235ba2e" (UID: "65fd0171-6ab8-4cb2-9801-d3444235ba2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:42:05.325460 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.325403    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65fd0171-6ab8-4cb2-9801-d3444235ba2e-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:05.325460 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.325426    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:05.325460 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.325436    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:05.325460 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.325445    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svvjf\" (UniqueName: \"kubernetes.io/projected/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kube-api-access-svvjf\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:05.325460 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.325453    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65fd0171-6ab8-4cb2-9801-d3444235ba2e-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:05.620508 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.620474    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.67:8643/healthz\": dial tcp 10.132.0.67:8643: connect: connection refused"
Sep 17 21:42:05.976981 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.976886    2586 generic.go:356] "Generic (PLEG): container finished" podID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerID="8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e" exitCode=0
Sep 17 21:42:05.976981 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.976952    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"
Sep 17 21:42:05.976981 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.976954    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerDied","Data":"8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e"}
Sep 17 21:42:05.977251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.977011    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg" event={"ID":"65fd0171-6ab8-4cb2-9801-d3444235ba2e","Type":"ContainerDied","Data":"3ce26ca5598d90521638e60837b34b5046998367c89be687334c656703803ed2"}
Sep 17 21:42:05.977251 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.977036    2586 scope.go:117] "RemoveContainer" containerID="d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a"
Sep 17 21:42:05.985417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.985396    2586 scope.go:117] "RemoveContainer" containerID="8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e"
Sep 17 21:42:05.992436 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.992418    2586 scope.go:117] "RemoveContainer" containerID="5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4"
Sep 17 21:42:05.998842 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.998821    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"]
Sep 17 21:42:05.999551 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.999535    2586 scope.go:117] "RemoveContainer" containerID="d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a"
Sep 17 21:42:05.999797 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:42:05.999781    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a\": container with ID starting with d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a not found: ID does not exist" containerID="d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a"
Sep 17 21:42:05.999862 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.999803    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a"} err="failed to get container status \"d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a\": rpc error: code = NotFound desc = could not find container \"d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a\": container with ID starting with d4dedc9cd3e79efeda6d4b5a300e8067258df4a1c363814e57beb742104b427a not found: ID does not exist"
Sep 17 21:42:05.999862 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:05.999817    2586 scope.go:117] "RemoveContainer" containerID="8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e"
Sep 17 21:42:06.000043 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:42:06.000026    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e\": container with ID starting with 8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e not found: ID does not exist" containerID="8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e"
Sep 17 21:42:06.000084 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.000049    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e"} err="failed to get container status \"8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e\": rpc error: code = NotFound desc = could not find container \"8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e\": container with ID starting with 8092a42f924ccb2969129ca26e624c357738e0bead2711c0d751cd641a0ef93e not found: ID does not exist"
Sep 17 21:42:06.000084 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.000064    2586 scope.go:117] "RemoveContainer" containerID="5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4"
Sep 17 21:42:06.000318 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:42:06.000302    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4\": container with ID starting with 5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4 not found: ID does not exist" containerID="5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4"
Sep 17 21:42:06.000446 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.000322    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4"} err="failed to get container status \"5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4\": rpc error: code = NotFound desc = could not find container \"5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4\": container with ID starting with 5709f7d6dff5cba4b3b1afb1d6ae3d574427989f89fdd6f3f669470a1aeaffb4 not found: ID does not exist"
Sep 17 21:42:06.002231 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.002211    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-nqbzg"]
Sep 17 21:42:06.541683 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.541647    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"]
Sep 17 21:42:06.541962 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.541945    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container"
Sep 17 21:42:06.542046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.541965    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container"
Sep 17 21:42:06.542046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.541986    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="storage-initializer"
Sep 17 21:42:06.542046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.541995    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="storage-initializer"
Sep 17 21:42:06.542046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542009    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2253c225-a592-4444-8708-223cae58e754" containerName="storage-initializer"
Sep 17 21:42:06.542046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542018    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2253c225-a592-4444-8708-223cae58e754" containerName="storage-initializer"
Sep 17 21:42:06.542046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542036    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kube-rbac-proxy"
Sep 17 21:42:06.542046 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542045    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kube-rbac-proxy"
Sep 17 21:42:06.542435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542111    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kube-rbac-proxy"
Sep 17 21:42:06.542435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542124    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2253c225-a592-4444-8708-223cae58e754" containerName="storage-initializer"
Sep 17 21:42:06.542435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542135    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" containerName="kserve-container"
Sep 17 21:42:06.542435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542212    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2253c225-a592-4444-8708-223cae58e754" containerName="storage-initializer"
Sep 17 21:42:06.542435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542221    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2253c225-a592-4444-8708-223cae58e754" containerName="storage-initializer"
Sep 17 21:42:06.542435 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.542281    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2253c225-a592-4444-8708-223cae58e754" containerName="storage-initializer"
Sep 17 21:42:06.546664 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.546644    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.548913 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.548895    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d588d2-predictor-serving-cert\""
Sep 17 21:42:06.549155 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.549137    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d588d2-kube-rbac-proxy-sar-config\""
Sep 17 21:42:06.554528 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.554504    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"]
Sep 17 21:42:06.634509 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.634478    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-cabundle-cert\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.634874 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.634517    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.634874 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.634579    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4badcf81-cd9b-42fe-935c-9aa056d836c1-kserve-provision-location\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.634874 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.634600    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4badcf81-cd9b-42fe-935c-9aa056d836c1-proxy-tls\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.634874 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.634655    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdgpn\" (UniqueName: \"kubernetes.io/projected/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-api-access-bdgpn\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.670833 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.670809    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65fd0171-6ab8-4cb2-9801-d3444235ba2e" path="/var/lib/kubelet/pods/65fd0171-6ab8-4cb2-9801-d3444235ba2e/volumes"
Sep 17 21:42:06.735625 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.735600    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.735625 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.735626    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4badcf81-cd9b-42fe-935c-9aa056d836c1-kserve-provision-location\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.735804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.735646    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4badcf81-cd9b-42fe-935c-9aa056d836c1-proxy-tls\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.735804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.735668    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdgpn\" (UniqueName: \"kubernetes.io/projected/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-api-access-bdgpn\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.735804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.735711    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-cabundle-cert\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.736183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.736157    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4badcf81-cd9b-42fe-935c-9aa056d836c1-kserve-provision-location\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.736307 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.736288    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.736374 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.736342    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-cabundle-cert\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.738029 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.738011    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4badcf81-cd9b-42fe-935c-9aa056d836c1-proxy-tls\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.743770 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.743750    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdgpn\" (UniqueName: \"kubernetes.io/projected/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-api-access-bdgpn\") pod \"isvc-primary-d588d2-predictor-cf985f74f-426n7\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") " pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:06.856909 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:06.856885    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:07.128604 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.128580    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"]
Sep 17 21:42:07.131012 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:42:07.130975    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4badcf81_cd9b_42fe_935c_9aa056d836c1.slice/crio-f55e3a25002507a41cb314bfc7b662928d987895735c4c1aad7a29d7fb8c7f3f WatchSource:0}: Error finding container f55e3a25002507a41cb314bfc7b662928d987895735c4c1aad7a29d7fb8c7f3f: Status 404 returned error can't find the container with id f55e3a25002507a41cb314bfc7b662928d987895735c4c1aad7a29d7fb8c7f3f
Sep 17 21:42:07.480625 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.480603    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:42:07.541893 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.541863    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-cabundle-cert\") pod \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") "
Sep 17 21:42:07.542032 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.541900    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls\") pod \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") "
Sep 17 21:42:07.542032 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.541933    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmcv6\" (UniqueName: \"kubernetes.io/projected/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-api-access-zmcv6\") pod \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") "
Sep 17 21:42:07.542032 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.541957    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kserve-provision-location\") pod \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") "
Sep 17 21:42:07.542205 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.542063    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-rbac-proxy-sar-config\") pod \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\" (UID: \"70b97cae-d5dc-41b9-9b4d-add78d57c31d\") "
Sep 17 21:42:07.542370 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.542247    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "70b97cae-d5dc-41b9-9b4d-add78d57c31d" (UID: "70b97cae-d5dc-41b9-9b4d-add78d57c31d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:42:07.542448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.542393    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "70b97cae-d5dc-41b9-9b4d-add78d57c31d" (UID: "70b97cae-d5dc-41b9-9b4d-add78d57c31d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:42:07.542448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.542418    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "70b97cae-d5dc-41b9-9b4d-add78d57c31d" (UID: "70b97cae-d5dc-41b9-9b4d-add78d57c31d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:42:07.544005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.543988    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "70b97cae-d5dc-41b9-9b4d-add78d57c31d" (UID: "70b97cae-d5dc-41b9-9b4d-add78d57c31d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:42:07.544005 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.543988    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-api-access-zmcv6" (OuterVolumeSpecName: "kube-api-access-zmcv6") pod "70b97cae-d5dc-41b9-9b4d-add78d57c31d" (UID: "70b97cae-d5dc-41b9-9b4d-add78d57c31d"). InnerVolumeSpecName "kube-api-access-zmcv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:42:07.643292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.643266    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/70b97cae-d5dc-41b9-9b4d-add78d57c31d-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:07.643292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.643287    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmcv6\" (UniqueName: \"kubernetes.io/projected/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-api-access-zmcv6\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:07.643292 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.643296    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:07.643682 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.643305    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:07.643682 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.643315    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/70b97cae-d5dc-41b9-9b4d-add78d57c31d-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:42:07.987903 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.987868    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerStarted","Data":"6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406"}
Sep 17 21:42:07.988060 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.987934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerStarted","Data":"f55e3a25002507a41cb314bfc7b662928d987895735c4c1aad7a29d7fb8c7f3f"}
Sep 17 21:42:07.989763 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.989738    2586 generic.go:356] "Generic (PLEG): container finished" podID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerID="06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2" exitCode=0
Sep 17 21:42:07.989890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.989772    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerDied","Data":"06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2"}
Sep 17 21:42:07.989890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.989779    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"
Sep 17 21:42:07.989890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.989826    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm" event={"ID":"70b97cae-d5dc-41b9-9b4d-add78d57c31d","Type":"ContainerDied","Data":"2190240a3501b7596f2b8dd75d1e334dadb600bb50efe67fbd33d4a0ac004e84"}
Sep 17 21:42:07.989890 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:07.989864    2586 scope.go:117] "RemoveContainer" containerID="0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a"
Sep 17 21:42:08.001441 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.001422    2586 scope.go:117] "RemoveContainer" containerID="06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2"
Sep 17 21:42:08.008945 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.008927    2586 scope.go:117] "RemoveContainer" containerID="5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98"
Sep 17 21:42:08.015841 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.015823    2586 scope.go:117] "RemoveContainer" containerID="0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a"
Sep 17 21:42:08.016097 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:42:08.016075    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a\": container with ID starting with 0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a not found: ID does not exist" containerID="0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a"
Sep 17 21:42:08.016137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.016104    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a"} err="failed to get container status \"0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a\": rpc error: code = NotFound desc = could not find container \"0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a\": container with ID starting with 0380ebed7115cb29aadcd9e58423b117b03cc494b65cf2a71ce2837d5869d48a not found: ID does not exist"
Sep 17 21:42:08.016137 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.016121    2586 scope.go:117] "RemoveContainer" containerID="06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2"
Sep 17 21:42:08.016440 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:42:08.016348    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2\": container with ID starting with 06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2 not found: ID does not exist" containerID="06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2"
Sep 17 21:42:08.016440 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.016398    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2"} err="failed to get container status \"06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2\": rpc error: code = NotFound desc = could not find container \"06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2\": container with ID starting with 06d79317024799f8a80e5a9ab19f1f2859b9932f890535ceba6606cf5e2db2e2 not found: ID does not exist"
Sep 17 21:42:08.016440 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.016419    2586 scope.go:117] "RemoveContainer" containerID="5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98"
Sep 17 21:42:08.016657 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:42:08.016641    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98\": container with ID starting with 5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98 not found: ID does not exist" containerID="5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98"
Sep 17 21:42:08.016704 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.016662    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98"} err="failed to get container status \"5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98\": rpc error: code = NotFound desc = could not find container \"5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98\": container with ID starting with 5d6fd58ee06cafa4459272cd2c1e0aeebc54e1b64c6290539c1fd381db6cff98 not found: ID does not exist"
Sep 17 21:42:08.019428 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.019406    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"]
Sep 17 21:42:08.023094 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.023072    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-wbczm"]
Sep 17 21:42:08.672259 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.672233    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" path="/var/lib/kubelet/pods/70b97cae-d5dc-41b9-9b4d-add78d57c31d/volumes"
Sep 17 21:42:08.734398 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734372    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"]
Sep 17 21:42:08.734609 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734598    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container"
Sep 17 21:42:08.734658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734611    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container"
Sep 17 21:42:08.734658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734625    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kube-rbac-proxy"
Sep 17 21:42:08.734658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734630    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kube-rbac-proxy"
Sep 17 21:42:08.734658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734642    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="storage-initializer"
Sep 17 21:42:08.734658 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734648    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="storage-initializer"
Sep 17 21:42:08.734804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734687    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kserve-container"
Sep 17 21:42:08.734804 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.734696    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b97cae-d5dc-41b9-9b4d-add78d57c31d" containerName="kube-rbac-proxy"
Sep 17 21:42:08.739255 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.739233    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.741724 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.741707    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 21:42:08.742348 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.742328    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 21:42:08.750794 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.750772    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"]
Sep 17 21:42:08.851734 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.851707    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.851869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.851748    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.851869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.851770    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.851869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.851811    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.851869 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.851850    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smhpt\" (UniqueName: \"kubernetes.io/projected/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-api-access-smhpt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.952516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.952444    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.952516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.952490    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.952516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.952508    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.952516 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.952523    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smhpt\" (UniqueName: \"kubernetes.io/projected/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-api-access-smhpt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.952805 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.952555    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.952974 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.952955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.953179 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.953158    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.953284 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.953268    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.955076 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.955055    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:08.971253 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:08.971231    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smhpt\" (UniqueName: \"kubernetes.io/projected/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-api-access-smhpt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:09.048484 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:09.048463    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:09.321040 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:09.320967    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"]
Sep 17 21:42:09.323791 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:42:09.323767    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6190b16c_0f32_4dd4_a000_e0a3076b8c0a.slice/crio-88adee18a029863c28767219dc85189b9abd0e94cdd1f88a7901422e5732721d WatchSource:0}: Error finding container 88adee18a029863c28767219dc85189b9abd0e94cdd1f88a7901422e5732721d: Status 404 returned error can't find the container with id 88adee18a029863c28767219dc85189b9abd0e94cdd1f88a7901422e5732721d
Sep 17 21:42:10.000202 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:10.000165    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerStarted","Data":"f8ec0ed5aa2d60f4bec4c0780eba05ad871ec5436943ebca1f821da1aa803770"}
Sep 17 21:42:10.000600 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:10.000226    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerStarted","Data":"88adee18a029863c28767219dc85189b9abd0e94cdd1f88a7901422e5732721d"}
Sep 17 21:42:12.008826 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:12.008796    2586 generic.go:356] "Generic (PLEG): container finished" podID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerID="6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406" exitCode=0
Sep 17 21:42:12.009271 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:12.008868    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerDied","Data":"6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406"}
Sep 17 21:42:13.014540 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:13.014510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerStarted","Data":"12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839"}
Sep 17 21:42:13.014930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:13.014573    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerStarted","Data":"412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8"}
Sep 17 21:42:13.014930 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:13.014776    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:13.038959 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:13.038893    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podStartSLOduration=3.063778761 podStartE2EDuration="7.038868048s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975089287s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:13.036753812 +0000 UTC m=+3148.570693621" watchObservedRunningTime="2026-09-17 21:42:13.038868048 +0000 UTC m=+3148.572807860"
Sep 17 21:42:14.020109 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:14.020077    2586 generic.go:356] "Generic (PLEG): container finished" podID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerID="f8ec0ed5aa2d60f4bec4c0780eba05ad871ec5436943ebca1f821da1aa803770" exitCode=0
Sep 17 21:42:14.020598 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:14.020153    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerDied","Data":"f8ec0ed5aa2d60f4bec4c0780eba05ad871ec5436943ebca1f821da1aa803770"}
Sep 17 21:42:14.020730 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:14.020704    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:14.022185 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:14.022163    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:42:15.025609 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:15.025564    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerStarted","Data":"3f13dc83169f33baa894c1987f952b304ea4034a34b3dcc120033b499190b782"}
Sep 17 21:42:15.026050 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:15.025630    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerStarted","Data":"adedbf08e0113015c0d4685c7a96267afc937ff4329f5a7fcdb14cd9dcb4ab46"}
Sep 17 21:42:15.026050 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:15.025869    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:42:15.026050 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:15.025952    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:15.049683 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:15.049642    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podStartSLOduration=3.312808155 podStartE2EDuration="7.04962753s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736819375s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:42:15.048350855 +0000 UTC m=+3150.582290691" watchObservedRunningTime="2026-09-17 21:42:15.04962753 +0000 UTC m=+3150.583567339"
Sep 17 21:42:16.029706 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:16.029675    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:16.031012 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:16.030981    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:42:17.033198 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:17.033157    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:42:20.032036 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:20.032004    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:42:20.032563 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:20.032499    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:42:22.037250 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:22.037223    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:42:22.037754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:22.037725    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:42:30.032834 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:30.032796    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:42:32.038135 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:32.038096    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:42:40.032990 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:40.032952    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:42:42.038284 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:42.038246    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:42:50.032764 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:50.032721    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:42:52.037641 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:42:52.037595    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:43:00.033096 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:00.033003    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:43:02.037823 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:02.037782    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:43:10.032578 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:10.032538    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:43:12.038336 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:12.038298    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:43:20.032989 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:20.032947    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:43:22.038657 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:22.038619    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:43:30.033619 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:30.033543    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:43:31.666375 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:31.666315    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:43:41.666537 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:41.666499    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:43:50.465743 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:50.465711    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"]
Sep 17 21:43:50.466171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:50.465975    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" containerID="cri-o://adedbf08e0113015c0d4685c7a96267afc937ff4329f5a7fcdb14cd9dcb4ab46" gracePeriod=30
Sep 17 21:43:50.466171 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:50.465996    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kube-rbac-proxy" containerID="cri-o://3f13dc83169f33baa894c1987f952b304ea4034a34b3dcc120033b499190b782" gracePeriod=30
Sep 17 21:43:51.372502 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:51.372468    2586 generic.go:356] "Generic (PLEG): container finished" podID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerID="3f13dc83169f33baa894c1987f952b304ea4034a34b3dcc120033b499190b782" exitCode=2
Sep 17 21:43:51.372752 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:51.372541    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerDied","Data":"3f13dc83169f33baa894c1987f952b304ea4034a34b3dcc120033b499190b782"}
Sep 17 21:43:51.666676 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:51.666575    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.69:8080: connect: connection refused"
Sep 17 21:43:52.034339 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:52.034243    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.69:8643/healthz\": dial tcp 10.132.0.69:8643: connect: connection refused"
Sep 17 21:43:54.386248 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.386219    2586 generic.go:356] "Generic (PLEG): container finished" podID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerID="adedbf08e0113015c0d4685c7a96267afc937ff4329f5a7fcdb14cd9dcb4ab46" exitCode=0
Sep 17 21:43:54.386595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.386256    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerDied","Data":"adedbf08e0113015c0d4685c7a96267afc937ff4329f5a7fcdb14cd9dcb4ab46"}
Sep 17 21:43:54.386595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.386302    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls" event={"ID":"6190b16c-0f32-4dd4-a000-e0a3076b8c0a","Type":"ContainerDied","Data":"88adee18a029863c28767219dc85189b9abd0e94cdd1f88a7901422e5732721d"}
Sep 17 21:43:54.386595 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.386322    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88adee18a029863c28767219dc85189b9abd0e94cdd1f88a7901422e5732721d"
Sep 17 21:43:54.395574 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.395555    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:43:54.519376 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519272    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smhpt\" (UniqueName: \"kubernetes.io/projected/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-api-access-smhpt\") pod \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") "
Sep 17 21:43:54.519376 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519315    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-cabundle-cert\") pod \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") "
Sep 17 21:43:54.519611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519400    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-rbac-proxy-sar-config\") pod \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") "
Sep 17 21:43:54.519611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519426    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-proxy-tls\") pod \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") "
Sep 17 21:43:54.519611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519458    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kserve-provision-location\") pod \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\" (UID: \"6190b16c-0f32-4dd4-a000-e0a3076b8c0a\") "
Sep 17 21:43:54.519778 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519733    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6190b16c-0f32-4dd4-a000-e0a3076b8c0a" (UID: "6190b16c-0f32-4dd4-a000-e0a3076b8c0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:43:54.519837 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519769    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6190b16c-0f32-4dd4-a000-e0a3076b8c0a" (UID: "6190b16c-0f32-4dd4-a000-e0a3076b8c0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:43:54.519837 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.519784    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6190b16c-0f32-4dd4-a000-e0a3076b8c0a" (UID: "6190b16c-0f32-4dd4-a000-e0a3076b8c0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:43:54.521456 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.521432    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-api-access-smhpt" (OuterVolumeSpecName: "kube-api-access-smhpt") pod "6190b16c-0f32-4dd4-a000-e0a3076b8c0a" (UID: "6190b16c-0f32-4dd4-a000-e0a3076b8c0a"). InnerVolumeSpecName "kube-api-access-smhpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:43:54.521515 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.521473    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6190b16c-0f32-4dd4-a000-e0a3076b8c0a" (UID: "6190b16c-0f32-4dd4-a000-e0a3076b8c0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:43:54.620028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.619987    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:43:54.620028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.620021    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:43:54.620028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.620032    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:43:54.620028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.620041    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:43:54.620281 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:54.620049    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smhpt\" (UniqueName: \"kubernetes.io/projected/6190b16c-0f32-4dd4-a000-e0a3076b8c0a-kube-api-access-smhpt\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:43:55.390250 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:55.390219    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"
Sep 17 21:43:55.408461 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:55.408435    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"]
Sep 17 21:43:55.412307 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:55.412284    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bpjls"]
Sep 17 21:43:56.670029 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:43:56.669997    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" path="/var/lib/kubelet/pods/6190b16c-0f32-4dd4-a000-e0a3076b8c0a/volumes"
Sep 17 21:44:09.783662 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:09.783627    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"]
Sep 17 21:44:09.784166 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:09.783851    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" containerID="cri-o://412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8" gracePeriod=30
Sep 17 21:44:09.784166 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:09.783893    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kube-rbac-proxy" containerID="cri-o://12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839" gracePeriod=30
Sep 17 21:44:10.026754 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:10.026719    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.68:8643/healthz\": dial tcp 10.132.0.68:8643: connect: connection refused"
Sep 17 21:44:10.033078 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:10.033057    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.68:8080: connect: connection refused"
Sep 17 21:44:10.446769 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:10.446738    2586 generic.go:356] "Generic (PLEG): container finished" podID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerID="12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839" exitCode=2
Sep 17 21:44:10.446938 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:10.446798    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerDied","Data":"12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839"}
Sep 17 21:44:14.120591 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.120565    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:44:14.259327 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259245    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-rbac-proxy-sar-config\") pod \"4badcf81-cd9b-42fe-935c-9aa056d836c1\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") "
Sep 17 21:44:14.259327 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259313    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdgpn\" (UniqueName: \"kubernetes.io/projected/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-api-access-bdgpn\") pod \"4badcf81-cd9b-42fe-935c-9aa056d836c1\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") "
Sep 17 21:44:14.259611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259334    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4badcf81-cd9b-42fe-935c-9aa056d836c1-proxy-tls\") pod \"4badcf81-cd9b-42fe-935c-9aa056d836c1\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") "
Sep 17 21:44:14.259611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259381    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4badcf81-cd9b-42fe-935c-9aa056d836c1-kserve-provision-location\") pod \"4badcf81-cd9b-42fe-935c-9aa056d836c1\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") "
Sep 17 21:44:14.259611 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259411    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-cabundle-cert\") pod \"4badcf81-cd9b-42fe-935c-9aa056d836c1\" (UID: \"4badcf81-cd9b-42fe-935c-9aa056d836c1\") "
Sep 17 21:44:14.259765 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259671    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4badcf81-cd9b-42fe-935c-9aa056d836c1" (UID: "4badcf81-cd9b-42fe-935c-9aa056d836c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:44:14.259765 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259692    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4badcf81-cd9b-42fe-935c-9aa056d836c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4badcf81-cd9b-42fe-935c-9aa056d836c1" (UID: "4badcf81-cd9b-42fe-935c-9aa056d836c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 21:44:14.259765 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.259718    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4badcf81-cd9b-42fe-935c-9aa056d836c1" (UID: "4badcf81-cd9b-42fe-935c-9aa056d836c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 21:44:14.261388 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.261371    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4badcf81-cd9b-42fe-935c-9aa056d836c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4badcf81-cd9b-42fe-935c-9aa056d836c1" (UID: "4badcf81-cd9b-42fe-935c-9aa056d836c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 21:44:14.261525 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.261510    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-api-access-bdgpn" (OuterVolumeSpecName: "kube-api-access-bdgpn") pod "4badcf81-cd9b-42fe-935c-9aa056d836c1" (UID: "4badcf81-cd9b-42fe-935c-9aa056d836c1"). InnerVolumeSpecName "kube-api-access-bdgpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 21:44:14.359957 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.359923    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4badcf81-cd9b-42fe-935c-9aa056d836c1-proxy-tls\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:44:14.359957 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.359954    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4badcf81-cd9b-42fe-935c-9aa056d836c1-kserve-provision-location\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:44:14.360148 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.359970    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-cabundle-cert\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:44:14.360148 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.359982    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:44:14.360148 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.359995    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdgpn\" (UniqueName: \"kubernetes.io/projected/4badcf81-cd9b-42fe-935c-9aa056d836c1-kube-api-access-bdgpn\") on node \"ip-10-0-143-254.ec2.internal\" DevicePath \"\""
Sep 17 21:44:14.462733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.462703    2586 generic.go:356] "Generic (PLEG): container finished" podID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerID="412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8" exitCode=0
Sep 17 21:44:14.462899 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.462758    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"
Sep 17 21:44:14.462899 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.462770    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerDied","Data":"412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8"}
Sep 17 21:44:14.462899 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.462817    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7" event={"ID":"4badcf81-cd9b-42fe-935c-9aa056d836c1","Type":"ContainerDied","Data":"f55e3a25002507a41cb314bfc7b662928d987895735c4c1aad7a29d7fb8c7f3f"}
Sep 17 21:44:14.462899 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.462842    2586 scope.go:117] "RemoveContainer" containerID="12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839"
Sep 17 21:44:14.470888 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.470864    2586 scope.go:117] "RemoveContainer" containerID="412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8"
Sep 17 21:44:14.477858 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.477844    2586 scope.go:117] "RemoveContainer" containerID="6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406"
Sep 17 21:44:14.483649 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.483629    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"]
Sep 17 21:44:14.485028 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.485013    2586 scope.go:117] "RemoveContainer" containerID="12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839"
Sep 17 21:44:14.485301 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:44:14.485279    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839\": container with ID starting with 12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839 not found: ID does not exist" containerID="12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839"
Sep 17 21:44:14.485389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.485313    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839"} err="failed to get container status \"12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839\": rpc error: code = NotFound desc = could not find container \"12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839\": container with ID starting with 12a7f7eef6b9696641c9aaa1161c2057aa053af28a16f00d9e24803a53712839 not found: ID does not exist"
Sep 17 21:44:14.485389 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.485337    2586 scope.go:117] "RemoveContainer" containerID="412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8"
Sep 17 21:44:14.485634 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:44:14.485615    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8\": container with ID starting with 412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8 not found: ID does not exist" containerID="412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8"
Sep 17 21:44:14.485674 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.485643    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8"} err="failed to get container status \"412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8\": rpc error: code = NotFound desc = could not find container \"412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8\": container with ID starting with 412d22559803b1677abe2e13419de42dea50a3e1764b696e9431c0529cae62d8 not found: ID does not exist"
Sep 17 21:44:14.485674 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.485660    2586 scope.go:117] "RemoveContainer" containerID="6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406"
Sep 17 21:44:14.485908 ip-10-0-143-254 kubenswrapper[2586]: E0917 21:44:14.485890    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406\": container with ID starting with 6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406 not found: ID does not exist" containerID="6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406"
Sep 17 21:44:14.485970 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.485917    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406"} err="failed to get container status \"6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406\": rpc error: code = NotFound desc = could not find container \"6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406\": container with ID starting with 6566982e6a7125aa2f2a7130722f9860afb866406cf1a9661dff7691e1003406 not found: ID does not exist"
Sep 17 21:44:14.488326 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.488305    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d588d2-predictor-cf985f74f-426n7"]
Sep 17 21:44:14.670374 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:14.670337    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" path="/var/lib/kubelet/pods/4badcf81-cd9b-42fe-935c-9aa056d836c1/volumes"
Sep 17 21:44:53.502002 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.501967    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4cltr/must-gather-zhwfx"]
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502198    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="storage-initializer"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502209    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="storage-initializer"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502220    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kube-rbac-proxy"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502226    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kube-rbac-proxy"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502235    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="storage-initializer"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502240    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="storage-initializer"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502246    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502252    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502260    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502266    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502273    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kube-rbac-proxy"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502278    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kube-rbac-proxy"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502320    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kserve-container"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502327    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4badcf81-cd9b-42fe-935c-9aa056d836c1" containerName="kube-rbac-proxy"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502334    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kserve-container"
Sep 17 21:44:53.502533 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.502340    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6190b16c-0f32-4dd4-a000-e0a3076b8c0a" containerName="kube-rbac-proxy"
Sep 17 21:44:53.505043 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.505024    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:53.507590 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.507560    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4cltr\"/\"default-dockercfg-9rqsm\""
Sep 17 21:44:53.507847 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.507829    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4cltr\"/\"kube-root-ca.crt\""
Sep 17 21:44:53.507924 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.507844    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4cltr\"/\"openshift-service-ca.crt\""
Sep 17 21:44:53.520047 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.520022    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtjk\" (UniqueName: \"kubernetes.io/projected/9016b4c7-8b57-4359-b665-07d794c153e1-kube-api-access-whtjk\") pod \"must-gather-zhwfx\" (UID: \"9016b4c7-8b57-4359-b665-07d794c153e1\") " pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:53.520154 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.520056    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9016b4c7-8b57-4359-b665-07d794c153e1-must-gather-output\") pod \"must-gather-zhwfx\" (UID: \"9016b4c7-8b57-4359-b665-07d794c153e1\") " pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:53.521779 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.521760    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cltr/must-gather-zhwfx"]
Sep 17 21:44:53.621022 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.620992    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9016b4c7-8b57-4359-b665-07d794c153e1-must-gather-output\") pod \"must-gather-zhwfx\" (UID: \"9016b4c7-8b57-4359-b665-07d794c153e1\") " pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:53.621210 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.621087    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtjk\" (UniqueName: \"kubernetes.io/projected/9016b4c7-8b57-4359-b665-07d794c153e1-kube-api-access-whtjk\") pod \"must-gather-zhwfx\" (UID: \"9016b4c7-8b57-4359-b665-07d794c153e1\") " pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:53.621307 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.621287    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9016b4c7-8b57-4359-b665-07d794c153e1-must-gather-output\") pod \"must-gather-zhwfx\" (UID: \"9016b4c7-8b57-4359-b665-07d794c153e1\") " pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:53.629295 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.629265    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtjk\" (UniqueName: \"kubernetes.io/projected/9016b4c7-8b57-4359-b665-07d794c153e1-kube-api-access-whtjk\") pod \"must-gather-zhwfx\" (UID: \"9016b4c7-8b57-4359-b665-07d794c153e1\") " pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:53.813825 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:53.813745    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cltr/must-gather-zhwfx"
Sep 17 21:44:54.087559 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:54.087536    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cltr/must-gather-zhwfx"]
Sep 17 21:44:54.089823 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:44:54.089794    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9016b4c7_8b57_4359_b665_07d794c153e1.slice/crio-bda8293ecdb2359bb58a246acc3d1982ce3defa9eb49208c6c897f7d1d2a5dee WatchSource:0}: Error finding container bda8293ecdb2359bb58a246acc3d1982ce3defa9eb49208c6c897f7d1d2a5dee: Status 404 returned error can't find the container with id bda8293ecdb2359bb58a246acc3d1982ce3defa9eb49208c6c897f7d1d2a5dee
Sep 17 21:44:54.091769 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:54.091751    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 21:44:54.606278 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:54.606245    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cltr/must-gather-zhwfx" event={"ID":"9016b4c7-8b57-4359-b665-07d794c153e1","Type":"ContainerStarted","Data":"bda8293ecdb2359bb58a246acc3d1982ce3defa9eb49208c6c897f7d1d2a5dee"}
Sep 17 21:44:55.617184 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:55.617146    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cltr/must-gather-zhwfx" event={"ID":"9016b4c7-8b57-4359-b665-07d794c153e1","Type":"ContainerStarted","Data":"386c08411e11cc80f4bd8488575cba4bad6711a5f2a2e5a26cda1cca8aeab320"}
Sep 17 21:44:55.617592 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:55.617194    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cltr/must-gather-zhwfx" event={"ID":"9016b4c7-8b57-4359-b665-07d794c153e1","Type":"ContainerStarted","Data":"524e874344ee9cb6d7ff70d89bc46b4eb40126ae368420d00b8861bbdbd05001"}
Sep 17 21:44:55.633733 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:55.633686    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4cltr/must-gather-zhwfx" podStartSLOduration=1.954850188 podStartE2EDuration="2.633668565s" totalImagesPullingTime="678.818377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 21:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:44:55.632797165 +0000 UTC m=+3311.166736969" watchObservedRunningTime="2026-09-17 21:44:55.633668565 +0000 UTC m=+3311.167608374"
Sep 17 21:44:56.453146 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:56.453115    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x4x76_32ea5724-e1d9-4592-8b7c-32e9d08b5388/global-pull-secret-syncer/0.log"
Sep 17 21:44:56.596598 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:56.596570    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rrt59_eac2d386-b829-49d2-ab71-42201ebfcf07/konnectivity-agent/0.log"
Sep 17 21:44:56.670581 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:44:56.670556    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-254.ec2.internal_0d17c601b9e0bec6e3d509315a4aa586/haproxy/0.log"
Sep 17 21:45:00.899417 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:00.899390    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcd5w_9081b054-8a9e-4345-af29-f353483d47f7/node-exporter/0.log"
Sep 17 21:45:00.923947 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:00.923913    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcd5w_9081b054-8a9e-4345-af29-f353483d47f7/kube-rbac-proxy/0.log"
Sep 17 21:45:00.950463 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:00.950390    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pcd5w_9081b054-8a9e-4345-af29-f353483d47f7/init-textfile/0.log"
Sep 17 21:45:04.452931 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.452898    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"]
Sep 17 21:45:04.458101 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.458073    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.468234 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.468207    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"]
Sep 17 21:45:04.505731 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.505697    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-proc\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.505897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.505749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqnlf\" (UniqueName: \"kubernetes.io/projected/723caca6-b51e-4202-bb22-be79f791144b-kube-api-access-kqnlf\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.505897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.505789    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-podres\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.505897 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.505854    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-sys\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.506025 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.505911    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-lib-modules\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607051 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607020    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqnlf\" (UniqueName: \"kubernetes.io/projected/723caca6-b51e-4202-bb22-be79f791144b-kube-api-access-kqnlf\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607051 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607055    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-podres\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-sys\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607106    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-lib-modules\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607142    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-proc\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607204    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-podres\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607218    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-proc\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607272 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607265    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-lib-modules\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.607508 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.607322    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/723caca6-b51e-4202-bb22-be79f791144b-sys\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.615070 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.615043    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqnlf\" (UniqueName: \"kubernetes.io/projected/723caca6-b51e-4202-bb22-be79f791144b-kube-api-access-kqnlf\") pod \"perf-node-gather-daemonset-2gt7q\" (UID: \"723caca6-b51e-4202-bb22-be79f791144b\") " pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:04.770523 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:04.770442    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:05.053900 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.053857    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"]
Sep 17 21:45:05.056608 ip-10-0-143-254 kubenswrapper[2586]: W0917 21:45:05.056579    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod723caca6_b51e_4202_bb22_be79f791144b.slice/crio-5ccd2c2028aad43253e58e9840c6e759541c58117d54c607d93456ae1e27b081 WatchSource:0}: Error finding container 5ccd2c2028aad43253e58e9840c6e759541c58117d54c607d93456ae1e27b081: Status 404 returned error can't find the container with id 5ccd2c2028aad43253e58e9840c6e759541c58117d54c607d93456ae1e27b081
Sep 17 21:45:05.399183 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.399146    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zc5rd_f03e58e0-ebc5-4501-b36f-a94873e29386/dns/0.log"
Sep 17 21:45:05.419481 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.419435    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zc5rd_f03e58e0-ebc5-4501-b36f-a94873e29386/kube-rbac-proxy/0.log"
Sep 17 21:45:05.489637 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.489609    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wlqs4_a2c6d687-7ce5-41d2-beea-c8e7833fc4e0/dns-node-resolver/0.log"
Sep 17 21:45:05.679929 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.679841    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q" event={"ID":"723caca6-b51e-4202-bb22-be79f791144b","Type":"ContainerStarted","Data":"62fb256fc8b551687737f127331c60e2349805e0778c889e8385c1a3c4329947"}
Sep 17 21:45:05.679929 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.679905    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q" event={"ID":"723caca6-b51e-4202-bb22-be79f791144b","Type":"ContainerStarted","Data":"5ccd2c2028aad43253e58e9840c6e759541c58117d54c607d93456ae1e27b081"}
Sep 17 21:45:05.680117 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.679946    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:05.696693 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:05.696636    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q" podStartSLOduration=1.696615871 podStartE2EDuration="1.696615871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 21:45:05.695786548 +0000 UTC m=+3321.229726359" watchObservedRunningTime="2026-09-17 21:45:05.696615871 +0000 UTC m=+3321.230555683"
Sep 17 21:45:06.005370 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:06.005280    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7c4kq_8f3930b3-92c5-4083-94fe-1881c5969abd/node-ca/0.log"
Sep 17 21:45:07.132043 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:07.132014    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8xwln_08af7dd3-8619-4aae-a156-128707928ac2/serve-healthcheck-canary/0.log"
Sep 17 21:45:07.668762 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:07.668738    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zpxgv_2757f4d0-ed48-4bca-81ab-d509f1a74567/kube-rbac-proxy/0.log"
Sep 17 21:45:07.691112 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:07.691087    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zpxgv_2757f4d0-ed48-4bca-81ab-d509f1a74567/exporter/0.log"
Sep 17 21:45:07.714986 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:07.714958    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zpxgv_2757f4d0-ed48-4bca-81ab-d509f1a74567/extractor/0.log"
Sep 17 21:45:10.044654 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.044625    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-b8mkq_a641e910-30bd-4176-995b-6ec545163e8e/manager/0.log"
Sep 17 21:45:10.376418 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.376385    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-qds2c_869396f8-52cc-46b9-a014-8f625e0311bf/manager/0.log"
Sep 17 21:45:10.408681 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.408658    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-st85t_a5913159-3c07-4321-a774-e6ac985e54df/s3-init/0.log"
Sep 17 21:45:10.423561 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.423542    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-st85t_a5913159-3c07-4321-a774-e6ac985e54df/download-hf-model/0.log"
Sep 17 21:45:10.448936 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.448916    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7hw5m_bb79a5af-bebd-40eb-8106-9ce2233eb6d2/s3-tls-init-custom/0.log"
Sep 17 21:45:10.464705 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.464683    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7hw5m_bb79a5af-bebd-40eb-8106-9ce2233eb6d2/download-hf-model/0.log"
Sep 17 21:45:10.491067 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.491044    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-46hb8_85de3a6f-cc0e-4ebe-9ece-c109813ba960/s3-tls-init-serving/0.log"
Sep 17 21:45:10.505927 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:10.505910    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-46hb8_85de3a6f-cc0e-4ebe-9ece-c109813ba960/download-hf-model/0.log"
Sep 17 21:45:11.701109 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:11.701075    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4cltr/perf-node-gather-daemonset-2gt7q"
Sep 17 21:45:16.527172 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.527145    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d85tw_5200bd98-2df0-453a-9ea0-251c997ede89/kube-multus-additional-cni-plugins/0.log"
Sep 17 21:45:16.547640 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.547614    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d85tw_5200bd98-2df0-453a-9ea0-251c997ede89/egress-router-binary-copy/0.log"
Sep 17 21:45:16.570236 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.570214    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d85tw_5200bd98-2df0-453a-9ea0-251c997ede89/cni-plugins/0.log"
Sep 17 21:45:16.589776 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.589753    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d85tw_5200bd98-2df0-453a-9ea0-251c997ede89/bond-cni-plugin/0.log"
Sep 17 21:45:16.609550 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.609529    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d85tw_5200bd98-2df0-453a-9ea0-251c997ede89/routeoverride-cni/0.log"
Sep 17 21:45:16.631466 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.631445    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d85tw_5200bd98-2df0-453a-9ea0-251c997ede89/whereabouts-cni-bincopy/0.log"
Sep 17 21:45:16.680029 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.680009    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d85tw_5200bd98-2df0-453a-9ea0-251c997ede89/whereabouts-cni/0.log"
Sep 17 21:45:16.919034 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.918971    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p8cpg_5174cf12-96bf-4526-b5ee-66dcc04371c0/kube-multus/0.log"
Sep 17 21:45:16.940498 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.940465    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2rvtb_0efd83dd-81e7-487a-b68e-2b9197c5f655/network-metrics-daemon/0.log"
Sep 17 21:45:16.959182 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:16.959164    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2rvtb_0efd83dd-81e7-487a-b68e-2b9197c5f655/kube-rbac-proxy/0.log"
Sep 17 21:45:18.304588 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.304559    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-controller/0.log"
Sep 17 21:45:18.322312 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.322293    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/1.log"
Sep 17 21:45:18.322443 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.322293    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovn-acl-logging/0.log"
Sep 17 21:45:18.340894 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.340874    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/kube-rbac-proxy-node/0.log"
Sep 17 21:45:18.364344 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.364325    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 21:45:18.385103 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.385079    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/northd/0.log"
Sep 17 21:45:18.406984 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.406963    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/nbdb/0.log"
Sep 17 21:45:18.431163 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.431148    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/sbdb/0.log"
Sep 17 21:45:18.483448 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:18.483414    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hg45d_818325ca-b623-4f67-8664-9e8d89e50ddc/ovnkube-controller/0.log"
Sep 17 21:45:19.591165 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:19.591135    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-v57r9_be6cd316-29b1-4d56-9ae4-f9688eba78ec/network-check-target-container/0.log"
Sep 17 21:45:20.641873 ip-10-0-143-254 kubenswrapper[2586]: I0917 21:45:20.641849    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-wjnf2_078730b6-ff30-4d6b-9146-451243e73b49/iptables-alerter/0.log"