Sep 17 17:35:08.707302 ip-10-0-132-143 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:35:09.049969 ip-10-0-132-143 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:35:09.049969 ip-10-0-132-143 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:35:09.049969 ip-10-0-132-143 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:35:09.049969 ip-10-0-132-143 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:35:09.049969 ip-10-0-132-143 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:35:09.050900 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.050811    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:35:09.054050 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054034    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:09.054050 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054049    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054054    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054058    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054061    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054064    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054067    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054070    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054073    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054075    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054078    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054088    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054091    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054094    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054096    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054099    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054102    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054105    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054110    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054114    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:09.054120 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054117    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054121    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054125    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054128    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054132    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054136    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054139    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054147    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054150    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054153    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054158    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054161    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054165    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054167    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054170    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054173    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054176    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054178    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054181    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:09.054641 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054184    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054186    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054189    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054192    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054195    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054197    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054200    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054203    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054205    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054208    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054211    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054214    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054216    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054219    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054222    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054225    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054229    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054232    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054234    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054237    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:09.055168 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054241    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054244    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054247    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054249    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054252    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054255    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054257    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054260    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054263    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054265    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054269    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054272    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054274    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054277    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054280    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054282    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054285    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054288    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054298    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:09.055682 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054301    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054305    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054308    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054312    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054314    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054317    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054320    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054322    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054326    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054329    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054331    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054334    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054337    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054339    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054342    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054345    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054348    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054351    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054354    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054356    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:09.056181 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.054359    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:09.056844 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056830    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:09.056844 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056843    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056847    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056850    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056854    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056859    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056863    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056866    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056876    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056880    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056885    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056895    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056898    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056901    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056904    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056908    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:09.056906 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056912    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056916    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056920    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056923    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056926    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056929    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056932    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056935    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056938    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056941    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056944    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056947    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056950    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056952    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056955    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056958    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056960    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056963    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056966    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056968    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:09.057362 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056971    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056974    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056977    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056979    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056983    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.056999    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057002    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057005    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057014    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057017    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057020    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057022    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057025    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057028    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057032    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057035    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057037    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057040    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057043    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057046    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:09.057889 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057049    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057051    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057054    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057057    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057060    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057063    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057066    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057068    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057071    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057074    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057077    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057080    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057083    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057086    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057089    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057091    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057094    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057097    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057099    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057102    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:09.058430 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057105    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057113    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057116    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057119    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057122    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057125    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057129    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057132    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057135    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057137    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057140    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057143    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057146    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057148    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057151    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057154    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057157    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057159    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057162    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057165    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:09.058951 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057168    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057171    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057173    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057619    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057625    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057628    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057631    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057635    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057640    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057643    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057648    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057651    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057655    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057659    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057662    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057666    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057670    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057673    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057677    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:09.059496 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057680    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057684    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057687    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057690    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057693    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057696    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057699    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057702    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057705    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057708    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057711    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057713    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057716    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057719    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057721    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057724    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057727    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057730    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057733    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:09.060051 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057737    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057741    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057745    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057748    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057751    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057753    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057756    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057759    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057763    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057766    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057769    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057771    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057775    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057778    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057781    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057783    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057786    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057789    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057792    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:09.060559 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057795    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057797    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057800    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057803    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057805    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057808    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057811    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057814    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057816    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057819    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057822    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057825    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057828    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057830    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057833    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057836    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057838    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057841    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057844    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057847    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:09.061076 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057849    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057852    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057856    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057858    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057861    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057864    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057867    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057871    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057874    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057877    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057880    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057882    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057885    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057888    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057890    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057893    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057896    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057899    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057902    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057905    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:09.061593 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057907    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057911    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057914    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057916    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.057919    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058003    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058040    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058047    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058052    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058056    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058060    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058064    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058068    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058072    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058075    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058080    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058083    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058087    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058091    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058094    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058097    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058100    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058103    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058109    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:35:09.062124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058112    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058115    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058119    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058122    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058126    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058129    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058132    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058136    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058140    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058144    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058147    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058151    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058154    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058159    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058162    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058165    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058169    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058172    2562 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058175    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058179    2562 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058182    2562 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058185    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058189    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058192    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:35:09.062741 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058200    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058206    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058210    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058213    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058216    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058219    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058222    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058226    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058229    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058232    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058235    2562 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058239    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058242    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058245    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058249    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058252    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058256    2562 flags.go:64] FLAG: --help="false"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058259    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058262    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058265    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058268    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058272    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058276    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058278    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:35:09.063390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058281    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058284    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058287    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058291    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058294    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058297    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058300    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058303    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058306    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058311    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058314    2562 flags.go:64] FLAG: --lock-file=""
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058317    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058320    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058324    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058329    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058332    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058335    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058338    2562 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058342    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058345    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058348    2562 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058351    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058356    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058359    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058363    2562 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:35:09.064063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058367    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058370    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058373    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058376    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058379    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058382    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058386    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058393    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058396    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058400    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058403    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058406    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058412    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058415    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058419    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058422    2562 flags.go:64] FLAG: --port="10250"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058425    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058431    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e3f0e86ef7243a2e"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058435    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058439    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058442    2562 flags.go:64] FLAG: --register-node="true"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058445    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058449    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058452    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:35:09.064721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058455    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058458    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058462    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058466    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058469    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058472    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058475    2562 flags.go:64] FLAG: --runonce="false"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058478    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058483    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058487    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058490    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058493    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058497    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058500    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058503    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058506    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058510    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058513    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058516    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058519    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058522    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058525    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058531    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058534    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058537    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:35:09.065408 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058543    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058547    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058550    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058553    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058556    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058559    2562 flags.go:64] FLAG: --v="2"
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058564    2562 flags.go:64] FLAG: --version="false"
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058568    2562 flags.go:64] FLAG: --vmodule=""
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058574    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.058578    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058694    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058698    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058701    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058704    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058707    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058712    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058715    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058717    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058721    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058723    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058726    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058729    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:09.066178 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058732    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058735    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058738    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058741    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058745    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058748    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058750    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058753    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058757    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058761    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058766    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058769    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058772    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058775    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058778    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058781    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058784    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058787    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058789    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058792    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:09.066771 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058795    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058798    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058801    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058804    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058807    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058811    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058814    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058817    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058820    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058822    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058825    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058828    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058831    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058834    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058836    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058839    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058842    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058845    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058848    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058850    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:09.067350 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058853    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058856    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058860    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058865    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058868    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058876    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058881    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058885    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058888    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058891    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058895    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058897    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058901    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058904    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058907    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058910    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058912    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058917    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058919    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:09.067871 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058922    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058925    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058928    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058931    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058934    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058936    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058939    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058942    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058945    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058948    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058951    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058954    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058957    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058960    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058963    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058967    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058970    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058972    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058975    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:09.068433 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058978    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058981    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.058998    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.059003    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.059007    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.059011    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.059015    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.059019    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.059022    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.059580    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.066777    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.066793    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.067463    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.067478    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067569    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:09.068931 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067574    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067578    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067582    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067586    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067589    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067591    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067594    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067597    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067600    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067603    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067606    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067609    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067612    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067614    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067617    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067620    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067623    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067627    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067631    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:09.069877 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067633    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067638    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067642    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067645    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067648    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067651    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067656    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067659    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067662    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067666    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067669    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067672    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067675    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067678    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067681    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067684    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067687    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067690    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067692    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067695    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:09.070388 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067698    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067701    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067704    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067706    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067709    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067712    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067715    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067718    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067720    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067723    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067726    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067736    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067743    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067746    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067748    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067751    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067755    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067757    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067760    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067763    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:09.070948 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067766    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067769    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067772    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067775    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067778    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067781    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067784    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067787    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067789    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067792    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067795    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067797    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067800    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067803    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067806    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067809    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067811    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067814    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067818    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:09.071572 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067822    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067825    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067828    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067831    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067833    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067842    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067845    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067848    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067851    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067854    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067857    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067860    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067862    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067865    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067868    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067871    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067874    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067876    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067879    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:09.072109 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.067883    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.067888    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068069    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068075    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068078    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068081    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068085    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068089    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068092    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068095    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068105    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068108    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068111    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068114    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068117    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:35:09.072598 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068120    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068122    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068125    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068128    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068137    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068140    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068143    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068146    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068148    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068151    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068154    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068156    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068159    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068162    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068165    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068169    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068173    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068176    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068180    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:35:09.073005 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068183    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068186    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068189    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068192    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068195    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068197    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068200    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068203    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068206    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068208    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068211    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068214    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068217    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068221    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068225    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068228    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068231    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068234    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068242    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068246    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:35:09.073492 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068249    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068252    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068254    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068257    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068260    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068263    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068266    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068269    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068271    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068275    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068279    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068282    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068285    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068288    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068291    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068293    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068296    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068299    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068302    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068304    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:35:09.074021 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068307    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068310    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068312    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068315    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068318    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068321    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068324    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068326    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068329    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068332    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068335    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068343    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068346    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068349    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068352    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068355    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068358    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068361    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068363    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:35:09.074539 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068366    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068369    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068372    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068375    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068377    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068380    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068383    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:09.068386    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.068391    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.068485    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.070764    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.071580    2562 server.go:1019] "Starting client certificate rotation"
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.071692    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:35:09.075089 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.072326    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:35:09.075434 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.074790    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:35:09.079218 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.079195    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:35:09.087964 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.087944    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:35:09.089623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.089606    2562 log.go:25] "Validated CRI v1 image API"
Sep 17 17:35:09.090557 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.090540    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:35:09.095660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.095638    2562 fs.go:135] Filesystem UUIDs: map[4e1102b8-c721-4384-98b5-af759e85ced8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c9eb5aac-6b1b-4177-96d6-c52bf9825ab3:/dev/nvme0n1p4]
Sep 17 17:35:09.095760 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.095658    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:35:09.101683 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.101551    2562 manager.go:216] Machine: {Timestamp:2026-09-17 17:35:09.10031307 +0000 UTC m=+0.287748913 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109406 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c5ff8b118cc4f3906206fec4467c SystemUUID:ec26c5ff-8b11-8cc4-f390-6206fec4467c BootID:7c06bb9e-ccf6-4da9-831d-7d2ba06fd987 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:5b:1a:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:5b:1a:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:fd:bd:53:e4:9a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:35:09.101683 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.101670    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:35:09.101864 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.101776    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:35:09.104442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.104412    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:35:09.104599 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.104446    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:35:09.105263 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.105252    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:35:09.105304 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.105265    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:35:09.105304 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.105274    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:35:09.106331 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.106320    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:35:09.107587 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.107576    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:35:09.113119 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113102    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:35:09.113240 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113230    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:35:09.113272 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113266    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:35:09.113311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113277    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:35:09.113311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113292    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:35:09.113311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113303    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:35:09.113429 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113313    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:35:09.113706 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113689    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:35:09.113793 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.113717    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:35:09.116384 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.116364    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:35:09.116716 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.116698    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:35:09.119316 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.119300    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:35:09.119383 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.119334    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:35:09.120395 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120382    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:35:09.120442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120399    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:35:09.120442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120406    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:35:09.120442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120413    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:35:09.120442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120422    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:35:09.120442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120427    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:35:09.120442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120443    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:35:09.120617 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120454    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:35:09.120617 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120470    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:35:09.120617 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120476    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:35:09.120617 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120489    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:35:09.120617 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.120495    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:35:09.121135 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.121123    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:35:09.121172 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.121141    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:35:09.124844 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.124831    2562 server.go:1268] "Started kubelet"
Sep 17 17:35:09.125015 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.124942    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:35:09.125015 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.124935    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:35:09.125127 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.125055    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:35:09.125692 ip-10-0-132-143 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:35:09.126863 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.126640    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:35:09.128361 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.128344    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:35:09.129902 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.129879    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:35:09.131104 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.131086    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:35:09.131457 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.131425    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:35:09.131538 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.131460    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:35:09.133404 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.131099    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-143.ec2.internal.18d62c353ecc1275  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-143.ec2.internal,UID:ip-10-0-132-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-143.ec2.internal,},FirstTimestamp:2026-09-17 17:35:09.124801141 +0000 UTC m=+0.312236984,LastTimestamp:2026-09-17 17:35:09.124801141 +0000 UTC m=+0.312236984,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-143.ec2.internal,}"
Sep 17 17:35:09.135822 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.135804    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:35:09.136593 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.136575    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:35:09.137126 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.137105    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.137455 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137434    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:35:09.137455 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137447    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:35:09.137455 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137458    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:35:09.137635 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137491    2562 factory.go:55] Registering systemd factory
Sep 17 17:35:09.137635 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137524    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:35:09.137635 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137588    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:35:09.137635 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137597    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:35:09.137819 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137778    2562 factory.go:153] Registering CRI-O factory
Sep 17 17:35:09.137819 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137792    2562 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:35:09.137919 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137858    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:35:09.137919 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137879    2562 factory.go:103] Registering Raw factory
Sep 17 17:35:09.137919 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.137896    2562 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:35:09.138395 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.138380    2562 manager.go:349] Starting recovery of all containers
Sep 17 17:35:09.139186 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.139152    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 17:35:09.139695 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.139653    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 17:35:09.145539 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.145504    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:35:09.149574 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.149552    2562 manager.go:354] Recovery completed
Sep 17 17:35:09.154413 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.154393    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:09.157191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.157161    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:09.157258 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.157204    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:09.157258 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.157223    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:09.157784 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.157768    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:35:09.157784 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.157782    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:35:09.157953 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.157786    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:09.157953 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.157801    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:35:09.158541 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.158525    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:09.158594 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.158559    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:09.158594 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.158575    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:09.160451 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.160436    2562 policy_none.go:49] "None policy: Start"
Sep 17 17:35:09.160528 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.160456    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:35:09.160528 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.160482    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:35:09.162895 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.162881    2562 policy_none.go:47] "Start"
Sep 17 17:35:09.172694 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.172626    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-143.ec2.internal.18d62c3540ba44ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-143.ec2.internal,UID:ip-10-0-132-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-143.ec2.internal,},FirstTimestamp:2026-09-17 17:35:09.15718878 +0000 UTC m=+0.344624622,LastTimestamp:2026-09-17 17:35:09.15718878 +0000 UTC m=+0.344624622,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-143.ec2.internal,}"
Sep 17 17:35:09.189835 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.189764    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-143.ec2.internal.18d62c3540baa005  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-143.ec2.internal,UID:ip-10-0-132-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-143.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-143.ec2.internal,},FirstTimestamp:2026-09-17 17:35:09.157212165 +0000 UTC m=+0.344648007,LastTimestamp:2026-09-17 17:35:09.157212165 +0000 UTC m=+0.344648007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-143.ec2.internal,}"
Sep 17 17:35:09.201169 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.201150    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x8x2q"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.201911    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-143.ec2.internal.18d62c3540bade3e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-143.ec2.internal,UID:ip-10-0-132-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-143.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-143.ec2.internal,},FirstTimestamp:2026-09-17 17:35:09.157228094 +0000 UTC m=+0.344663937,LastTimestamp:2026-09-17 17:35:09.157228094 +0000 UTC m=+0.344663937,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-143.ec2.internal,}"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.203747    2562 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.203822    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.203835    2562 server.go:87] "Starting device plugin registration server"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.204127    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.204141    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.204719    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.204822    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.204831    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.205744    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.205813    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.212816    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x8x2q"
Sep 17 17:35:09.220791 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.215272    2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-143.ec2.internal.18d62c3540ba44ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-143.ec2.internal.18d62c3540ba44ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-143.ec2.internal,UID:ip-10-0-132-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-143.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-143.ec2.internal,},FirstTimestamp:2026-09-17 17:35:09.15718878 +0000 UTC m=+0.344624622,LastTimestamp:2026-09-17 17:35:09.158542177 +0000 UTC m=+0.345978024,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-143.ec2.internal,}"
Sep 17 17:35:09.270874 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.270852    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:35:09.271011 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.270905    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:35:09.271011 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.270925    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:35:09.271011 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.270961    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:35:09.274351 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.274331    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:09.306010 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.305943    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:09.306962 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.306946    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:09.307045 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.306977    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:09.307045 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.306988    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:09.307045 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.307012    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.327379 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.327361    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.327446 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.327381    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-143.ec2.internal\": node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.354611 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.354596    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.371469 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.371442    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"]
Sep 17 17:35:09.371530 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.371512    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:09.373166 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.373147    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:09.373255 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.373184    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:09.373255 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.373197    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:09.375701 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.375688    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:09.375856 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.375840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.375896 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.375873    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:09.376547 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.376529    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:09.376632 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.376560    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:09.376632 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.376570    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:09.376632 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.376530    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:09.376732 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.376637    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:09.376732 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.376656    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:09.379096 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.379079    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.379173 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.379105    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:09.379907 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.379889    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:09.380023 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.379935    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:09.380023 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.379951    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:09.406749 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.406727    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-143.ec2.internal\" not found" node="ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.410281 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.410267    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-143.ec2.internal\" not found" node="ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.438921 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.438900    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c7975fab11cb60f6a1435abc617c9f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal\" (UID: \"3c7975fab11cb60f6a1435abc617c9f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.438986 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.438945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c7975fab11cb60f6a1435abc617c9f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal\" (UID: \"3c7975fab11cb60f6a1435abc617c9f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.438986 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.438965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68d557cea98b36ce4a2697ffb1cea14a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-143.ec2.internal\" (UID: \"68d557cea98b36ce4a2697ffb1cea14a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.455159 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.455136    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.539585 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.539554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c7975fab11cb60f6a1435abc617c9f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal\" (UID: \"3c7975fab11cb60f6a1435abc617c9f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.539585 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.539587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c7975fab11cb60f6a1435abc617c9f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal\" (UID: \"3c7975fab11cb60f6a1435abc617c9f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.539756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.539610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68d557cea98b36ce4a2697ffb1cea14a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-143.ec2.internal\" (UID: \"68d557cea98b36ce4a2697ffb1cea14a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.539756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.539663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c7975fab11cb60f6a1435abc617c9f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal\" (UID: \"3c7975fab11cb60f6a1435abc617c9f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.539756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.539671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68d557cea98b36ce4a2697ffb1cea14a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-143.ec2.internal\" (UID: \"68d557cea98b36ce4a2697ffb1cea14a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.539756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.539671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c7975fab11cb60f6a1435abc617c9f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal\" (UID: \"3c7975fab11cb60f6a1435abc617c9f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.555490 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.555469    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.656473 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.656407    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.710607 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.710573    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.714010 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.713994    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:09.757314 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.757158    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.858164 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.858123    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.959013 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:09.958939    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:09.967174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:09.967156    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:10.059838 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:10.059809    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:10.071842 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.071817    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:35:10.071983 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.071966    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:35:10.072026 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.071988    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:35:10.160020 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:10.159988    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:10.218959 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.218867    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:30:10 +0000 UTC" deadline="2028-04-15 07:51:47.294234512 +0000 UTC"
Sep 17 17:35:10.218959 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.218897    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13814h16m37.07534143s"
Sep 17 17:35:10.260997 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:10.260958    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-143.ec2.internal\" not found"
Sep 17 17:35:10.265130 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.265107    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:35:10.266623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.266599    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:35:10.266716 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.266642    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:35:10.266716 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.266659    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:35:10.325533 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.325503    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:10.337929 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.337876    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:10.359541 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.359523    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:35:10.361186 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.361172    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal"
Sep 17 17:35:10.376659 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:10.376628    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d557cea98b36ce4a2697ffb1cea14a.slice/crio-c0d90b84baabf6a7bf92b1a069332f33b781dca1441bee664bc7051ea5c78604 WatchSource:0}: Error finding container c0d90b84baabf6a7bf92b1a069332f33b781dca1441bee664bc7051ea5c78604: Status 404 returned error can't find the container with id c0d90b84baabf6a7bf92b1a069332f33b781dca1441bee664bc7051ea5c78604
Sep 17 17:35:10.377206 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:10.377186    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7975fab11cb60f6a1435abc617c9f3.slice/crio-103f8444aa9c0fa9bd9b2c9766bf10472b0ebe0d69bf6404016140611a0d3026 WatchSource:0}: Error finding container 103f8444aa9c0fa9bd9b2c9766bf10472b0ebe0d69bf6404016140611a0d3026: Status 404 returned error can't find the container with id 103f8444aa9c0fa9bd9b2c9766bf10472b0ebe0d69bf6404016140611a0d3026
Sep 17 17:35:10.382343 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.382325    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:35:10.423729 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.423704    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:35:10.634732 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:10.634701    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:11.117077 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.117038    2562 apiserver.go:52] "Watching apiserver"
Sep 17 17:35:11.125439 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.125412    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:35:11.126731 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.126698    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-nwhfq","openshift-multus/network-metrics-daemon-95wb9","openshift-network-diagnostics/network-check-target-dk7v5","openshift-network-operator/iptables-alerter-qfld2","openshift-ovn-kubernetes/ovnkube-node-qkrtg","kube-system/global-pull-secret-syncer-n9dx8","kube-system/konnectivity-agent-vbpgr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw","openshift-cluster-node-tuning-operator/tuned-9mp8q","openshift-dns/node-resolver-txcs7","openshift-image-registry/node-ca-z4zxd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal","openshift-multus/multus-additional-cni-plugins-w52g2","kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal"]
Sep 17 17:35:11.129648 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.129554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.132694 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.132673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.135824 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.135800    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:11.135952 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.135924    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:11.136442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.136422    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ncrnk\""
Sep 17 17:35:11.136522 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.136477    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:35:11.136686 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.136669    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:35:11.136773 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.136754    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lxj26\""
Sep 17 17:35:11.137058 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.137013    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:35:11.137945 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.137915    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:35:11.138379 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.138353    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.139457 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.139428    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:35:11.139568 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.139466    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:35:11.139912 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.139892    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:35:11.140298 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.139758    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:35:11.141080 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.140916    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:35:11.141924 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.141902    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f9fsm\""
Sep 17 17:35:11.144443 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.144424    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.145656 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.145633    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:35:11.145852 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.145833    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:35:11.147900 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.147459    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:35:11.147900 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.147653    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:35:11.147900 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.147813    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:35:11.148153 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-system-cni-dir\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.148153 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148077    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65c94\" (UniqueName: \"kubernetes.io/projected/48fdef06-8806-4e8d-bc0c-794bdb713240-kube-api-access-65c94\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.148153 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148109    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:11.148317 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad3032d4-e211-4afe-b702-2f06b5841986-konnectivity-ca\") pod \"konnectivity-agent-vbpgr\" (UID: \"ad3032d4-e211-4afe-b702-2f06b5841986\") " pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.148317 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-cnibin\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.148317 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-cni-binary-copy\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.148317 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.148531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.148531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f32820d-0d24-47a3-a833-8701f0844ab6-host-slash\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.148531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cm5\" (UniqueName: \"kubernetes.io/projected/4f32820d-0d24-47a3-a833-8701f0844ab6-kube-api-access-n5cm5\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.148531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad3032d4-e211-4afe-b702-2f06b5841986-agent-certs\") pod \"konnectivity-agent-vbpgr\" (UID: \"ad3032d4-e211-4afe-b702-2f06b5841986\") " pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.148531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.148531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148457    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f32820d-0d24-47a3-a833-8701f0844ab6-iptables-alerter-script\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.148531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148485    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:35:11.148916 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148600    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:35:11.148916 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.148811    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:35:11.149455 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.149258    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.149455 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.149348    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:11.149619 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.149478    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g5qvn\""
Sep 17 17:35:11.151951 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.151930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.154485 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.154466    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ngq6c\""
Sep 17 17:35:11.155563 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.155544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.156723 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.156704    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:35:11.156812 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.156746    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:35:11.157871 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.157842    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:11.157982 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.157916    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:11.160184 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.160162    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ml89q\""
Sep 17 17:35:11.160836 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.160808    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.161340 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.161185    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:35:11.163759 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.163083    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.166379 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.165601    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.169098 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.168983    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:35:11.169098 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.168998    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:35:11.169252 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.169203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:35:11.169347 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.169328    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:35:11.169419 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.169392    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:35:11.169669 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.169652    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:35:11.169965 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.169945    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:35:11.170061 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.169986    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:35:11.170061 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.170043    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:35:11.170174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.170086    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tqqbm\""
Sep 17 17:35:11.170307 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.170285    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s8nlb\""
Sep 17 17:35:11.170383 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.170315    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9q944\""
Sep 17 17:35:11.201621 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.201598    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zmjz9"
Sep 17 17:35:11.222694 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.222673    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zmjz9"
Sep 17 17:35:11.238511 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.238465    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:35:11.248751 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-cni-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.248872 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-etc-kubernetes\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.248872 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-run-netns\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.248872 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-node-log\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249066 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.249066 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-hosts-file\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.249066 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-cni-binary-copy\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.249066 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.248995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-slash\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249066 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-sys-fs\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysctl-conf\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad3032d4-e211-4afe-b702-2f06b5841986-agent-certs\") pod \"konnectivity-agent-vbpgr\" (UID: \"ad3032d4-e211-4afe-b702-2f06b5841986\") " pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-cni-bin\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249169    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6badc097-8f99-421a-a042-3e1991f71fc0-host\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-tmp\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249220    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-systemd\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-systemd\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-system-cni-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.249326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-kubelet\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-systemd-units\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-cni-netd\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysctl-d\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-daemon-config\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g74b6\" (UniqueName: \"kubernetes.io/projected/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-kube-api-access-g74b6\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-cni-binary-copy\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249479    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twx5c\" (UniqueName: \"kubernetes.io/projected/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-kube-api-access-twx5c\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.249870 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249537    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cm5\" (UniqueName: \"kubernetes.io/projected/4f32820d-0d24-47a3-a833-8701f0844ab6-kube-api-access-n5cm5\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.249968    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-tuned\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-conf-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250113    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250127    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6badc097-8f99-421a-a042-3e1991f71fc0-serviceca\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-sys\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250157    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-host\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmmt\" (UniqueName: \"kubernetes.io/projected/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-kube-api-access-vkmmt\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250205    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-cni-bin\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65c94\" (UniqueName: \"kubernetes.io/projected/48fdef06-8806-4e8d-bc0c-794bdb713240-kube-api-access-65c94\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovn-node-metrics-cert\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-modprobe-d\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.250354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-kubernetes\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-kubelet\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-multus-certs\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-log-socket\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250459    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad3032d4-e211-4afe-b702-2f06b5841986-konnectivity-ca\") pod \"konnectivity-agent-vbpgr\" (UID: \"ad3032d4-e211-4afe-b702-2f06b5841986\") " pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-etc-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ad3032d4-e211-4afe-b702-2f06b5841986-konnectivity-ca\") pod \"konnectivity-agent-vbpgr\" (UID: \"ad3032d4-e211-4afe-b702-2f06b5841986\") " pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.250960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-ovn\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovnkube-script-lib\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nmr6\" (UniqueName: \"kubernetes.io/projected/c596612c-3d0e-4d0d-a402-81ea054b0aa2-kube-api-access-5nmr6\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-cni-binary-copy\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-k8s-cni-cncf-io\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f32820d-0d24-47a3-a833-8701f0844ab6-host-slash\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251121    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f32820d-0d24-47a3-a833-8701f0844ab6-host-slash\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovnkube-config\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-cni-multus\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.251229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7q5d\" (UniqueName: \"kubernetes.io/projected/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-kube-api-access-l7q5d\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f32820d-0d24-47a3-a833-8701f0844ab6-iptables-alerter-script\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysconfig\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251321    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-var-lib-kubelet\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89sqk\" (UniqueName: \"kubernetes.io/projected/20234e60-7bf9-440e-8c44-7006a3c67a8f-kube-api-access-89sqk\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-cnibin\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f32820d-0d24-47a3-a833-8701f0844ab6-iptables-alerter-script\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251898    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-cnibin\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.251982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-cnibin\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252038    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-socket-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8dfece24-349b-48df-84f2-5b78d5c7feb5-dbus\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.252161 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-env-overrides\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-registration-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-socket-dir-parent\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-netns\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-tmp-dir\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252278    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-var-lib-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252302    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78kv\" (UniqueName: \"kubernetes.io/projected/6badc097-8f99-421a-a042-3e1991f71fc0-kube-api-access-j78kv\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-device-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8dfece24-349b-48df-84f2-5b78d5c7feb5-kubelet-config\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-hostroot\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-system-cni-dir\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-run\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-lib-modules\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/48fdef06-8806-4e8d-bc0c-794bdb713240-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.253016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.252619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/48fdef06-8806-4e8d-bc0c-794bdb713240-system-cni-dir\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.255599 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.255580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ad3032d4-e211-4afe-b702-2f06b5841986-agent-certs\") pod \"konnectivity-agent-vbpgr\" (UID: \"ad3032d4-e211-4afe-b702-2f06b5841986\") " pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.261387 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.261366    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:11.261486 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.261393    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:11.261486 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.261406    2562 projected.go:196] Error preparing data for projected volume kube-api-access-xqzpw for pod openshift-network-diagnostics/network-check-target-dk7v5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:11.261601 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.261494    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw podName:9bba983c-c289-4cbe-a15f-5e3877032bdd nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.76147362 +0000 UTC m=+2.948909468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xqzpw" (UniqueName: "kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw") pod "network-check-target-dk7v5" (UID: "9bba983c-c289-4cbe-a15f-5e3877032bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:11.267901 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.267866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65c94\" (UniqueName: \"kubernetes.io/projected/48fdef06-8806-4e8d-bc0c-794bdb713240-kube-api-access-65c94\") pod \"multus-additional-cni-plugins-w52g2\" (UID: \"48fdef06-8806-4e8d-bc0c-794bdb713240\") " pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.276844 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.276804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal" event={"ID":"68d557cea98b36ce4a2697ffb1cea14a","Type":"ContainerStarted","Data":"c0d90b84baabf6a7bf92b1a069332f33b781dca1441bee664bc7051ea5c78604"}
Sep 17 17:35:11.278827 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.278797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal" event={"ID":"3c7975fab11cb60f6a1435abc617c9f3","Type":"ContainerStarted","Data":"103f8444aa9c0fa9bd9b2c9766bf10472b0ebe0d69bf6404016140611a0d3026"}
Sep 17 17:35:11.286642 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.286613    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cm5\" (UniqueName: \"kubernetes.io/projected/4f32820d-0d24-47a3-a833-8701f0844ab6-kube-api-access-n5cm5\") pod \"iptables-alerter-qfld2\" (UID: \"4f32820d-0d24-47a3-a833-8701f0844ab6\") " pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.324587 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.324553    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:11.353254 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-var-lib-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.353391 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78kv\" (UniqueName: \"kubernetes.io/projected/6badc097-8f99-421a-a042-3e1991f71fc0-kube-api-access-j78kv\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.353391 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-device-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.353391 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8dfece24-349b-48df-84f2-5b78d5c7feb5-kubelet-config\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.353391 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-hostroot\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.353391 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-run\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-lib-modules\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-cni-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-etc-kubernetes\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-run-netns\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-node-log\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-hosts-file\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-slash\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.353660 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-sys-fs\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysctl-conf\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-run\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-cni-bin\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6badc097-8f99-421a-a042-3e1991f71fc0-host\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-tmp\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-systemd\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-var-lib-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353854    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-systemd\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-system-cni-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.353956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-kubelet\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-systemd-units\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-cni-netd\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysctl-d\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354080    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-daemon-config\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g74b6\" (UniqueName: \"kubernetes.io/projected/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-kube-api-access-g74b6\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:11.354191 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twx5c\" (UniqueName: \"kubernetes.io/projected/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-kube-api-access-twx5c\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-device-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-tuned\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-conf-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354288    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6badc097-8f99-421a-a042-3e1991f71fc0-serviceca\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354312    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8dfece24-349b-48df-84f2-5b78d5c7feb5-kubelet-config\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354315    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-sys\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-host\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-hostroot\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354381    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmmt\" (UniqueName: \"kubernetes.io/projected/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-kube-api-access-vkmmt\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354407    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-cni-bin\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovn-node-metrics-cert\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-modprobe-d\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.354463    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-kubernetes\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.355050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-kubelet\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.354531    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret podName:8dfece24-349b-48df-84f2-5b78d5c7feb5 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.854510078 +0000 UTC m=+3.041945928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret") pod "global-pull-secret-syncer-n9dx8" (UID: "8dfece24-349b-48df-84f2-5b78d5c7feb5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-kubelet\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-multus-certs\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-log-socket\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-multus-certs\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-etc-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-ovn\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-systemd\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovnkube-script-lib\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmr6\" (UniqueName: \"kubernetes.io/projected/c596612c-3d0e-4d0d-a402-81ea054b0aa2-kube-api-access-5nmr6\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354801    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-systemd-units\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-cni-binary-copy\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-k8s-cni-cncf-io\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-cni-netd\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovnkube-config\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354919    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-cni-multus\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.355904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7q5d\" (UniqueName: \"kubernetes.io/projected/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-kube-api-access-l7q5d\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysctl-d\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysconfig\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-var-lib-kubelet\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355078    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89sqk\" (UniqueName: \"kubernetes.io/projected/20234e60-7bf9-440e-8c44-7006a3c67a8f-kube-api-access-89sqk\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-cnibin\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-socket-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355191    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8dfece24-349b-48df-84f2-5b78d5c7feb5-dbus\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-env-overrides\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-registration-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-socket-dir-parent\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-netns\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-tmp-dir\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-daemon-config\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-tmp-dir\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.356796 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.355874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-lib-modules\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-cni-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-etc-kubernetes\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356122    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-run-netns\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356174    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-node-log\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356262    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-etc-selinux\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-hosts-file\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-slash\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-sys-fs\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-modprobe-d\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysctl-conf\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-kubernetes\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356622    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-sys\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356649    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6badc097-8f99-421a-a042-3e1991f71fc0-host\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-cni-bin\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356666    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-host\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-conf-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.357724 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.356775    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.356837    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:11.856820121 +0000 UTC m=+3.044255965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-sysconfig\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-var-lib-kubelet\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-etc-openvswitch\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-ovn\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.356981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6badc097-8f99-421a-a042-3e1991f71fc0-serviceca\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8dfece24-349b-48df-84f2-5b78d5c7feb5-dbus\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357076    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-var-lib-cni-multus\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357153    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357238    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-socket-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-log-socket\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357313    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-cnibin\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.354750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-system-cni-dir\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-kubelet\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357517    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-host-cni-bin\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.358756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-env-overrides\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20234e60-7bf9-440e-8c44-7006a3c67a8f-run-systemd\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-netns\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-multus-socket-dir-parent\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c596612c-3d0e-4d0d-a402-81ea054b0aa2-registration-dir\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357627    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-host-run-k8s-cni-cncf-io\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.357742    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovnkube-config\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.358068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovnkube-script-lib\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.358281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-cni-binary-copy\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.359677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.358878    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-etc-tuned\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.360368 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.360346    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-tmp\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.360460 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.360416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20234e60-7bf9-440e-8c44-7006a3c67a8f-ovn-node-metrics-cert\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.372466 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.372351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78kv\" (UniqueName: \"kubernetes.io/projected/6badc097-8f99-421a-a042-3e1991f71fc0-kube-api-access-j78kv\") pod \"node-ca-z4zxd\" (UID: \"6badc097-8f99-421a-a042-3e1991f71fc0\") " pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.372466 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.372394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g74b6\" (UniqueName: \"kubernetes.io/projected/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-kube-api-access-g74b6\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:11.373050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.372697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twx5c\" (UniqueName: \"kubernetes.io/projected/7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f-kube-api-access-twx5c\") pod \"node-resolver-txcs7\" (UID: \"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f\") " pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.374758 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.374734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmr6\" (UniqueName: \"kubernetes.io/projected/c596612c-3d0e-4d0d-a402-81ea054b0aa2-kube-api-access-5nmr6\") pod \"aws-ebs-csi-driver-node-mqbxw\" (UID: \"c596612c-3d0e-4d0d-a402-81ea054b0aa2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.375559 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.375498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmmt\" (UniqueName: \"kubernetes.io/projected/993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c-kube-api-access-vkmmt\") pod \"tuned-9mp8q\" (UID: \"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c\") " pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.376850 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.376826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89sqk\" (UniqueName: \"kubernetes.io/projected/20234e60-7bf9-440e-8c44-7006a3c67a8f-kube-api-access-89sqk\") pod \"ovnkube-node-qkrtg\" (UID: \"20234e60-7bf9-440e-8c44-7006a3c67a8f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.377481 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.377397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7q5d\" (UniqueName: \"kubernetes.io/projected/a927a7c0-1fd8-4c09-ba41-c5bea5579bd0-kube-api-access-l7q5d\") pod \"multus-nwhfq\" (UID: \"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0\") " pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.448548 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.448516    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:11.457675 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.457644    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w52g2"
Sep 17 17:35:11.469096 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.469047    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qfld2"
Sep 17 17:35:11.476149 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.476130    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:11.485605 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.485587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9mp8q"
Sep 17 17:35:11.495520 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.495498    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwhfq"
Sep 17 17:35:11.503398 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.503369    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-txcs7"
Sep 17 17:35:11.512132 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.512104    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z4zxd"
Sep 17 17:35:11.518945 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.518909    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw"
Sep 17 17:35:11.803431 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.803399    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:35:11.859486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.859448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:11.859685 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.859525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:11.859685 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:11.859586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:11.859685 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859622    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:11.859848 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859701    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:12.859679904 +0000 UTC m=+4.047115751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:11.859848 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859702    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:11.859848 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859711    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:11.859848 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859735    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:11.859848 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859750    2562 projected.go:196] Error preparing data for projected volume kube-api-access-xqzpw for pod openshift-network-diagnostics/network-check-target-dk7v5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:11.859848 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859759    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret podName:8dfece24-349b-48df-84f2-5b78d5c7feb5 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:12.85974207 +0000 UTC m=+4.047177916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret") pod "global-pull-secret-syncer-n9dx8" (UID: "8dfece24-349b-48df-84f2-5b78d5c7feb5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:11.859848 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:11.859797    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw podName:9bba983c-c289-4cbe-a15f-5e3877032bdd nodeName:}" failed. No retries permitted until 2026-09-17 17:35:12.859782398 +0000 UTC m=+4.047218226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqzpw" (UniqueName: "kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw") pod "network-check-target-dk7v5" (UID: "9bba983c-c289-4cbe-a15f-5e3877032bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:12.226389 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:12.226299    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:30:11 +0000 UTC" deadline="2028-06-09 02:29:36.857990468 +0000 UTC"
Sep 17 17:35:12.226389 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:12.226337    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15128h54m24.631658976s"
Sep 17 17:35:12.272176 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:12.272141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:12.272328 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.272260    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:12.411873 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.411848    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc596612c_3d0e_4d0d_a402_81ea054b0aa2.slice/crio-8722ec98ad92815c7b9bf29a3ea923e228864596276882ae3e7dcf03c87f9e0a WatchSource:0}: Error finding container 8722ec98ad92815c7b9bf29a3ea923e228864596276882ae3e7dcf03c87f9e0a: Status 404 returned error can't find the container with id 8722ec98ad92815c7b9bf29a3ea923e228864596276882ae3e7dcf03c87f9e0a
Sep 17 17:35:12.437604 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.437575    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f32820d_0d24_47a3_a833_8701f0844ab6.slice/crio-65e6f50c56e990d69a07b0b98a0a71ac80df814797b1286caf24307286162a39 WatchSource:0}: Error finding container 65e6f50c56e990d69a07b0b98a0a71ac80df814797b1286caf24307286162a39: Status 404 returned error can't find the container with id 65e6f50c56e990d69a07b0b98a0a71ac80df814797b1286caf24307286162a39
Sep 17 17:35:12.438872 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.438853    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3032d4_e211_4afe_b702_2f06b5841986.slice/crio-d90e248b1f0ca0035d3049b5162dcf62ad8f898f455dc029744ba5b62169680a WatchSource:0}: Error finding container d90e248b1f0ca0035d3049b5162dcf62ad8f898f455dc029744ba5b62169680a: Status 404 returned error can't find the container with id d90e248b1f0ca0035d3049b5162dcf62ad8f898f455dc029744ba5b62169680a
Sep 17 17:35:12.439758 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.439728    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6badc097_8f99_421a_a042_3e1991f71fc0.slice/crio-d0a4f6b5424cf509c497a450a50a21c3b81086b81d4ac52ede744d246b5a5224 WatchSource:0}: Error finding container d0a4f6b5424cf509c497a450a50a21c3b81086b81d4ac52ede744d246b5a5224: Status 404 returned error can't find the container with id d0a4f6b5424cf509c497a450a50a21c3b81086b81d4ac52ede744d246b5a5224
Sep 17 17:35:12.445930 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.445904    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fdef06_8806_4e8d_bc0c_794bdb713240.slice/crio-7038a839721bc034872fa2bc42d95ba9971346cf3dfb1cb177621f09d644920b WatchSource:0}: Error finding container 7038a839721bc034872fa2bc42d95ba9971346cf3dfb1cb177621f09d644920b: Status 404 returned error can't find the container with id 7038a839721bc034872fa2bc42d95ba9971346cf3dfb1cb177621f09d644920b
Sep 17 17:35:12.446896 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.446869    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20234e60_7bf9_440e_8c44_7006a3c67a8f.slice/crio-8085a2bda89e93bd2666d7b150b073cf41941fb518d33a7e7ac3a476cf50fae9 WatchSource:0}: Error finding container 8085a2bda89e93bd2666d7b150b073cf41941fb518d33a7e7ac3a476cf50fae9: Status 404 returned error can't find the container with id 8085a2bda89e93bd2666d7b150b073cf41941fb518d33a7e7ac3a476cf50fae9
Sep 17 17:35:12.448114 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.447950    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993100f5_fb4c_41f5_b6c3_e8ac26c4fe4c.slice/crio-2cf1de3f12fd6aad27b6be8846191b93133a25ffc83731f74703a65f140bf21c WatchSource:0}: Error finding container 2cf1de3f12fd6aad27b6be8846191b93133a25ffc83731f74703a65f140bf21c: Status 404 returned error can't find the container with id 2cf1de3f12fd6aad27b6be8846191b93133a25ffc83731f74703a65f140bf21c
Sep 17 17:35:12.449640 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.449557    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a655d6d_1a91_41e1_9ef1_4b3b3f7abe5f.slice/crio-c773b1b222a2bf4517e6c834465e75b09740cad40e5488da374d4ffd45b2d4fe WatchSource:0}: Error finding container c773b1b222a2bf4517e6c834465e75b09740cad40e5488da374d4ffd45b2d4fe: Status 404 returned error can't find the container with id c773b1b222a2bf4517e6c834465e75b09740cad40e5488da374d4ffd45b2d4fe
Sep 17 17:35:12.451335 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:12.451315    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda927a7c0_1fd8_4c09_ba41_c5bea5579bd0.slice/crio-31414ac16e08623f4e7a3521c37b80ae92ec3934c530659dbad906c41983f4d6 WatchSource:0}: Error finding container 31414ac16e08623f4e7a3521c37b80ae92ec3934c530659dbad906c41983f4d6: Status 404 returned error can't find the container with id 31414ac16e08623f4e7a3521c37b80ae92ec3934c530659dbad906c41983f4d6
Sep 17 17:35:12.868744 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:12.868659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:12.868744 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:12.868721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:12.868762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.868859    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.868908    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.868928    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.869016    2562 projected.go:196] Error preparing data for projected volume kube-api-access-xqzpw for pod openshift-network-diagnostics/network-check-target-dk7v5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.868955    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret podName:8dfece24-349b-48df-84f2-5b78d5c7feb5 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:14.868916989 +0000 UTC m=+6.056352828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret") pod "global-pull-secret-syncer-n9dx8" (UID: "8dfece24-349b-48df-84f2-5b78d5c7feb5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.869084    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw podName:9bba983c-c289-4cbe-a15f-5e3877032bdd nodeName:}" failed. No retries permitted until 2026-09-17 17:35:14.869066875 +0000 UTC m=+6.056502705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqzpw" (UniqueName: "kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw") pod "network-check-target-dk7v5" (UID: "9bba983c-c289-4cbe-a15f-5e3877032bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.868996    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:12.869227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:12.869126    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:14.86911561 +0000 UTC m=+6.056551439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:13.226928 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.226805    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:30:11 +0000 UTC" deadline="2028-07-03 10:07:43.808700546 +0000 UTC"
Sep 17 17:35:13.226928 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.226848    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15712h32m30.581857141s"
Sep 17 17:35:13.277537 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.276582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:13.277537 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:13.276772    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:13.277537 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.277340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:13.277537 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:13.277482    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:13.302233 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.302193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal" event={"ID":"68d557cea98b36ce4a2697ffb1cea14a","Type":"ContainerStarted","Data":"b82e73e91e213fe9e22fec36ec349868c58ec98e4d4c5efa2a2cf0931dc15466"}
Sep 17 17:35:13.307488 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.307456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwhfq" event={"ID":"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0","Type":"ContainerStarted","Data":"31414ac16e08623f4e7a3521c37b80ae92ec3934c530659dbad906c41983f4d6"}
Sep 17 17:35:13.311152 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.311127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mp8q" event={"ID":"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c","Type":"ContainerStarted","Data":"2cf1de3f12fd6aad27b6be8846191b93133a25ffc83731f74703a65f140bf21c"}
Sep 17 17:35:13.329934 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.329905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerStarted","Data":"7038a839721bc034872fa2bc42d95ba9971346cf3dfb1cb177621f09d644920b"}
Sep 17 17:35:13.342393 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.342363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbpgr" event={"ID":"ad3032d4-e211-4afe-b702-2f06b5841986","Type":"ContainerStarted","Data":"d90e248b1f0ca0035d3049b5162dcf62ad8f898f455dc029744ba5b62169680a"}
Sep 17 17:35:13.356273 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.356245    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qfld2" event={"ID":"4f32820d-0d24-47a3-a833-8701f0844ab6","Type":"ContainerStarted","Data":"65e6f50c56e990d69a07b0b98a0a71ac80df814797b1286caf24307286162a39"}
Sep 17 17:35:13.361606 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.361578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw" event={"ID":"c596612c-3d0e-4d0d-a402-81ea054b0aa2","Type":"ContainerStarted","Data":"8722ec98ad92815c7b9bf29a3ea923e228864596276882ae3e7dcf03c87f9e0a"}
Sep 17 17:35:13.365795 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.364969    2562 generic.go:356] "Generic (PLEG): container finished" podID="3c7975fab11cb60f6a1435abc617c9f3" containerID="16fa19ff912a562a64050a8493aeb36c0b117770b1d8e25ef2d43a99b102036d" exitCode=0
Sep 17 17:35:13.365795 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.365053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal" event={"ID":"3c7975fab11cb60f6a1435abc617c9f3","Type":"ContainerDied","Data":"16fa19ff912a562a64050a8493aeb36c0b117770b1d8e25ef2d43a99b102036d"}
Sep 17 17:35:13.374746 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.374715    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txcs7" event={"ID":"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f","Type":"ContainerStarted","Data":"c773b1b222a2bf4517e6c834465e75b09740cad40e5488da374d4ffd45b2d4fe"}
Sep 17 17:35:13.385809 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.385783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"8085a2bda89e93bd2666d7b150b073cf41941fb518d33a7e7ac3a476cf50fae9"}
Sep 17 17:35:13.393238 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.390824    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-143.ec2.internal" podStartSLOduration=2.390807121 podStartE2EDuration="2.390807121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:13.319963977 +0000 UTC m=+4.507399829" watchObservedRunningTime="2026-09-17 17:35:13.390807121 +0000 UTC m=+4.578242974"
Sep 17 17:35:13.407569 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:13.407504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4zxd" event={"ID":"6badc097-8f99-421a-a042-3e1991f71fc0","Type":"ContainerStarted","Data":"d0a4f6b5424cf509c497a450a50a21c3b81086b81d4ac52ede744d246b5a5224"}
Sep 17 17:35:14.271824 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:14.271788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:14.272390 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.271954    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:14.443015 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:14.442903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal" event={"ID":"3c7975fab11cb60f6a1435abc617c9f3","Type":"ContainerStarted","Data":"fad5acfb1d8b425addd1c0d9759ea81ac0b9d840149428092cbb705c5afd90c5"}
Sep 17 17:35:14.889789 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:14.889747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:14.889973 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:14.889816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:14.889973 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:14.889849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:14.890150 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890124    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:14.890213 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890139    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:14.890213 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890205    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:14.890309 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890229    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret podName:8dfece24-349b-48df-84f2-5b78d5c7feb5 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:18.89020638 +0000 UTC m=+10.077642228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret") pod "global-pull-secret-syncer-n9dx8" (UID: "8dfece24-349b-48df-84f2-5b78d5c7feb5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:14.890309 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890263    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:18.89024476 +0000 UTC m=+10.077680593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:14.890309 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890150    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:14.890309 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890286    2562 projected.go:196] Error preparing data for projected volume kube-api-access-xqzpw for pod openshift-network-diagnostics/network-check-target-dk7v5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:14.890524 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:14.890322    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw podName:9bba983c-c289-4cbe-a15f-5e3877032bdd nodeName:}" failed. No retries permitted until 2026-09-17 17:35:18.890311238 +0000 UTC m=+10.077747081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqzpw" (UniqueName: "kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw") pod "network-check-target-dk7v5" (UID: "9bba983c-c289-4cbe-a15f-5e3877032bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:15.273713 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:15.272081    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:15.273713 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:15.272230    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:15.274422 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:15.274400    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:15.274579 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:15.274552    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:16.272198 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:16.272162    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:16.272400 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:16.272321    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:17.271682 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:17.271629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:17.271682 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:17.271661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:17.272282 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:17.271796    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:17.272282 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:17.271904    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:18.272231 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:18.272187    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:18.272686 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.272337    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:18.923507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:18.923583    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:18.923640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.923701    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.923795    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret podName:8dfece24-349b-48df-84f2-5b78d5c7feb5 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:26.923771267 +0000 UTC m=+18.111207112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret") pod "global-pull-secret-syncer-n9dx8" (UID: "8dfece24-349b-48df-84f2-5b78d5c7feb5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.923848    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.923873    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.923888    2562 projected.go:196] Error preparing data for projected volume kube-api-access-xqzpw for pod openshift-network-diagnostics/network-check-target-dk7v5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:18.923989 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.923948    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw podName:9bba983c-c289-4cbe-a15f-5e3877032bdd nodeName:}" failed. No retries permitted until 2026-09-17 17:35:26.923930272 +0000 UTC m=+18.111366147 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqzpw" (UniqueName: "kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw") pod "network-check-target-dk7v5" (UID: "9bba983c-c289-4cbe-a15f-5e3877032bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:18.924591 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.924045    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:18.924591 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:18.924084    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:26.924069477 +0000 UTC m=+18.111505322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:19.271342 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:19.271283    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:19.272930 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:19.272320    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:19.272930 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:19.272445    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:19.272930 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:19.272876    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:20.272309 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:20.271779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:20.272309 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:20.271937    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:21.271722 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:21.271648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:21.272198 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:21.271811    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:21.272198 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:21.271697    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:21.272317 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:21.272219    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:22.271983 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:22.271953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:22.272420 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:22.272081    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:23.271707 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:23.271675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:23.271887 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:23.271778    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:23.271887 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:23.271847    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:23.272025 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:23.271938    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:24.272099 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:24.272055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:24.272513 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:24.272175    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:25.272232 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:25.272137    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:25.272706 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:25.272270    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:25.272706 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:25.272348    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:25.272706 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:25.272433    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:26.271639 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:26.271607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:26.271897 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.271745    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:26.988783 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:26.988745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:26.988817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:26.988867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.988888    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.988976    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.988990    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:42.988967254 +0000 UTC m=+34.176403096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.989001    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.989021    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret podName:8dfece24-349b-48df-84f2-5b78d5c7feb5 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:42.989009349 +0000 UTC m=+34.176445192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret") pod "global-pull-secret-syncer-n9dx8" (UID: "8dfece24-349b-48df-84f2-5b78d5c7feb5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.989027    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.989040    2562 projected.go:196] Error preparing data for projected volume kube-api-access-xqzpw for pod openshift-network-diagnostics/network-check-target-dk7v5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:26.989259 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:26.989096    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw podName:9bba983c-c289-4cbe-a15f-5e3877032bdd nodeName:}" failed. No retries permitted until 2026-09-17 17:35:42.989081315 +0000 UTC m=+34.176517143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqzpw" (UniqueName: "kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw") pod "network-check-target-dk7v5" (UID: "9bba983c-c289-4cbe-a15f-5e3877032bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:27.272139 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:27.272042    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:27.272139 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:27.272098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:27.272350 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:27.272196    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:27.272350 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:27.272308    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:28.272042 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:28.272011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:28.272480 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:28.272129    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:29.272553 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:29.272513    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:29.273017 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:29.272628    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:29.273017 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:29.272718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:29.273017 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:29.272811    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:30.271949 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:30.271902    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:30.272215 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:30.272030    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:31.271636 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.271442    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:31.272439 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.271537    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:31.272439 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:31.271731    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:31.272439 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:31.271833    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:31.481814 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.481711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwhfq" event={"ID":"a927a7c0-1fd8-4c09-ba41-c5bea5579bd0","Type":"ContainerStarted","Data":"f3fa4c0998424587e762cb483fbe4c26757d62e0db585ef91759c4e576bdcaae"}
Sep 17 17:35:31.483366 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.483331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9mp8q" event={"ID":"993100f5-fb4c-41f5-b6c3-e8ac26c4fe4c","Type":"ContainerStarted","Data":"363deb78e9aeb8d7aaa3c9618a032a4c6fabb83f975fc8a2e6f61ca692f36237"}
Sep 17 17:35:31.484816 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.484793    2562 generic.go:356] "Generic (PLEG): container finished" podID="48fdef06-8806-4e8d-bc0c-794bdb713240" containerID="589df363e3b7b0c6e9c594922d0228935b4dc6c323f98be1bcbbd9294aef0e14" exitCode=0
Sep 17 17:35:31.484932 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.484873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerDied","Data":"589df363e3b7b0c6e9c594922d0228935b4dc6c323f98be1bcbbd9294aef0e14"}
Sep 17 17:35:31.486321 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.486301    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbpgr" event={"ID":"ad3032d4-e211-4afe-b702-2f06b5841986","Type":"ContainerStarted","Data":"ea802123b38bbdf61310fc5421086f45ece08a5b72eb702bb8357fc4bb2f7cf1"}
Sep 17 17:35:31.487882 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.487865    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw" event={"ID":"c596612c-3d0e-4d0d-a402-81ea054b0aa2","Type":"ContainerStarted","Data":"0aa7888f3168cf5327f0acdeae3a1a4c76afcba3c3192ad99f24e6383b4549e2"}
Sep 17 17:35:31.489290 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.489266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-txcs7" event={"ID":"7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f","Type":"ContainerStarted","Data":"873b85249ad1f5d653b37a959f9433c4576ebdc05f32b282c9a9ac76c9e7c2b4"}
Sep 17 17:35:31.492190 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.492168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"6b0bde747eb93a3fc603315e486161a1e0ebe135919ff3768a6e8fcabb211972"}
Sep 17 17:35:31.492264 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.492220    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"12226bf98ffbde10a6165e52245acf3599bb71dcc01a37d65314aa4a50a70fd6"}
Sep 17 17:35:31.492304 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.492263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"5434004b3cec1af6b9b90177995d105e482ed6313214a18225fe97ed50a07cab"}
Sep 17 17:35:31.492304 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.492293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"ec2fcee5f01dd8d6b3cef6f6bae93bbcabbb1e802090be36d1f583ea18209801"}
Sep 17 17:35:31.492372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.492325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"8ee7056d57d5e71e31ad04936fb622ab8bb9fa25a24ddb5de2be6f879a8f058f"}
Sep 17 17:35:31.492372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.492356    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"5785df147725ec742b46401fe0a3d28cd2eef4c1a60923b6539841a0845a57dd"}
Sep 17 17:35:31.493463 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.493444    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z4zxd" event={"ID":"6badc097-8f99-421a-a042-3e1991f71fc0","Type":"ContainerStarted","Data":"25ab3f4a71a9f3ae5d20f6d199c3943e8c12571feee184b7a5386a14976338b9"}
Sep 17 17:35:31.516082 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.516044    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-143.ec2.internal" podStartSLOduration=20.516033433 podStartE2EDuration="20.516033433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:14.468982352 +0000 UTC m=+5.656418204" watchObservedRunningTime="2026-09-17 17:35:31.516033433 +0000 UTC m=+22.703469282"
Sep 17 17:35:31.538505 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.538465    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nwhfq" podStartSLOduration=3.439886565 podStartE2EDuration="21.538452845s" totalImagesPullingTime="18.09856628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:31.516319653 +0000 UTC m=+22.703755502" watchObservedRunningTime="2026-09-17 17:35:31.538452845 +0000 UTC m=+22.725888740"
Sep 17 17:35:31.563053 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.563019    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vbpgr" podStartSLOduration=7.30287515 podStartE2EDuration="21.563007744s" totalImagesPullingTime="14.260132594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:31.538584967 +0000 UTC m=+22.726020806" watchObservedRunningTime="2026-09-17 17:35:31.563007744 +0000 UTC m=+22.750443593"
Sep 17 17:35:31.603202 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.603162    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9mp8q" podStartSLOduration=3.567810238 podStartE2EDuration="21.603148301s" totalImagesPullingTime="18.035338063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:31.586147801 +0000 UTC m=+22.773583688" watchObservedRunningTime="2026-09-17 17:35:31.603148301 +0000 UTC m=+22.790584149"
Sep 17 17:35:31.634265 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.634227    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-txcs7" podStartSLOduration=3.602542652 podStartE2EDuration="21.634216634s" totalImagesPullingTime="18.031673982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:31.603599238 +0000 UTC m=+22.791035087" watchObservedRunningTime="2026-09-17 17:35:31.634216634 +0000 UTC m=+22.821652483"
Sep 17 17:35:31.634373 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.634289    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z4zxd" podStartSLOduration=3.592645011 podStartE2EDuration="21.634283968s" totalImagesPullingTime="18.041638957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:31.63406446 +0000 UTC m=+22.821500309" watchObservedRunningTime="2026-09-17 17:35:31.634283968 +0000 UTC m=+22.821719817"
Sep 17 17:35:31.930354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:31.930325    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:35:32.220022 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.219771    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:35:31.930352926Z","UUID":"9dfb2e5f-7a04-402f-81fe-24a997adcf91","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:35:32.221645 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.221624    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:35:32.221769 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.221655    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:35:32.271137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.271110    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:32.271297 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:32.271230    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:32.498756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.498721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qfld2" event={"ID":"4f32820d-0d24-47a3-a833-8701f0844ab6","Type":"ContainerStarted","Data":"5b67694e71087124b266a01b22de449b3ecfc5970f64e6d5e01d7463450afc05"}
Sep 17 17:35:32.501702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.501667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw" event={"ID":"c596612c-3d0e-4d0d-a402-81ea054b0aa2","Type":"ContainerStarted","Data":"85fa60f85510d4f01dfd3743fa9adaafc20b7c6892a3729e3753ed99f5c24f92"}
Sep 17 17:35:32.526129 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.526056    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qfld2" podStartSLOduration=4.484372062 podStartE2EDuration="22.526036709s" totalImagesPullingTime="18.041664647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:32.525402193 +0000 UTC m=+23.712838043" watchObservedRunningTime="2026-09-17 17:35:32.526036709 +0000 UTC m=+23.713472559"
Sep 17 17:35:32.765074 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:32.764999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:33.004869 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.004839    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:33.005441 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.005417    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:33.271687 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.271661    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:33.271897 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:33.271788    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:33.271897 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.271662    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:33.272049 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:33.271960    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:33.506134 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.506100    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw" event={"ID":"c596612c-3d0e-4d0d-a402-81ea054b0aa2","Type":"ContainerStarted","Data":"2854e0ca58d982dc27f62c5fccb4b6ec5d090b9dde5e1a9a5526fe2b727e2342"}
Sep 17 17:35:33.510170 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.510142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"344235c8effa1bc6008419852c0d63ccfca9933cb3a7daa453415bb983625b89"}
Sep 17 17:35:33.511000 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.510975    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbpgr"
Sep 17 17:35:33.526222 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:33.526180    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mqbxw" podStartSLOduration=3.445165539 podStartE2EDuration="23.526166819s" totalImagesPullingTime="20.08100128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:33.526148915 +0000 UTC m=+24.713584764" watchObservedRunningTime="2026-09-17 17:35:33.526166819 +0000 UTC m=+24.713602667"
Sep 17 17:35:34.271573 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:34.271533    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:34.271738 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:34.271655    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:35.272171 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:35.272139    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:35.272586 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:35.272241    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:35.272586 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:35.272269    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:35.272586 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:35.272324    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:35.521732 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:35.521519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" event={"ID":"20234e60-7bf9-440e-8c44-7006a3c67a8f","Type":"ContainerStarted","Data":"3d7a5386048edb4e723d58b0517de909aac5df511261df62984ce882023282c2"}
Sep 17 17:35:36.271745 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.271713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:36.271995 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:36.271820    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:36.524798 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.524727    2562 generic.go:356] "Generic (PLEG): container finished" podID="48fdef06-8806-4e8d-bc0c-794bdb713240" containerID="78ddf7a9a9926ba14ed02e2457325c0a86816c985e635c1d827f369574bad438" exitCode=0
Sep 17 17:35:36.524798 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.524784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerDied","Data":"78ddf7a9a9926ba14ed02e2457325c0a86816c985e635c1d827f369574bad438"}
Sep 17 17:35:36.525576 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.525231    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:36.525576 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.525305    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:36.525576 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.525350    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:36.539732 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.539709    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:36.539979 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.539904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:35:36.550041 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:36.550005    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg" podStartSLOduration=8.457717599 podStartE2EDuration="26.549993158s" totalImagesPullingTime="18.092275559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:35.551270514 +0000 UTC m=+26.738706374" watchObservedRunningTime="2026-09-17 17:35:36.549993158 +0000 UTC m=+27.737429040"
Sep 17 17:35:37.271444 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.271418    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:37.271621 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:37.271546    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:37.271621 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.271592    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:37.271745 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:37.271677    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:37.440764 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.439208    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dk7v5"]
Sep 17 17:35:37.442514 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.442489    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-95wb9"]
Sep 17 17:35:37.444925 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.444888    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9dx8"]
Sep 17 17:35:37.445033 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.445021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:37.445141 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:37.445116    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:37.527340 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.527154    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:37.527340 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:37.527162    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:37.527684 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:37.527653    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:37.527790 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:37.527769    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:38.531393 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:38.531363    2562 generic.go:356] "Generic (PLEG): container finished" podID="48fdef06-8806-4e8d-bc0c-794bdb713240" containerID="0b74e016069ca0bfc8adfa37ccf29bd9a871d65b7456f267454214f5e437af56" exitCode=0
Sep 17 17:35:38.531729 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:38.531443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerDied","Data":"0b74e016069ca0bfc8adfa37ccf29bd9a871d65b7456f267454214f5e437af56"}
Sep 17 17:35:39.272693 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:39.272658    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:39.272837 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:39.272774    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:39.272925 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:39.272829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:39.273025 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:39.272978    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:39.273025 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:39.272998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:39.273150 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:39.273092    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:39.535589 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:39.535551    2562 generic.go:356] "Generic (PLEG): container finished" podID="48fdef06-8806-4e8d-bc0c-794bdb713240" containerID="159ad62a2d1a0dad2a63cea49349da2d0fe4daa4335e0566b17b9b602dd882b0" exitCode=0
Sep 17 17:35:39.535945 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:39.535600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerDied","Data":"159ad62a2d1a0dad2a63cea49349da2d0fe4daa4335e0566b17b9b602dd882b0"}
Sep 17 17:35:41.271976 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:41.271944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:41.272511 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:41.271990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:41.272511 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:41.271944    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:41.272511 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:41.272062    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dk7v5" podUID="9bba983c-c289-4cbe-a15f-5e3877032bdd"
Sep 17 17:35:41.272511 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:41.272159    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n9dx8" podUID="8dfece24-349b-48df-84f2-5b78d5c7feb5"
Sep 17 17:35:41.272511 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:41.272244    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:35:43.004428 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.004393    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.004450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.004496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004569    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004658    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004679    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004686    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret podName:8dfece24-349b-48df-84f2-5b78d5c7feb5 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:15.00466147 +0000 UTC m=+66.192097367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret") pod "global-pull-secret-syncer-n9dx8" (UID: "8dfece24-349b-48df-84f2-5b78d5c7feb5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004698    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004712    2562 projected.go:196] Error preparing data for projected volume kube-api-access-xqzpw for pod openshift-network-diagnostics/network-check-target-dk7v5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004722    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:15.004703455 +0000 UTC m=+66.192139305 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:35:43.004854 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.004756    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw podName:9bba983c-c289-4cbe-a15f-5e3877032bdd nodeName:}" failed. No retries permitted until 2026-09-17 17:36:15.004741809 +0000 UTC m=+66.192177637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqzpw" (UniqueName: "kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw") pod "network-check-target-dk7v5" (UID: "9bba983c-c289-4cbe-a15f-5e3877032bdd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:35:43.045715 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.045693    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-143.ec2.internal" event="NodeReady"
Sep 17 17:35:43.045849 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.045834    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:35:43.176045 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.176018    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-545f7f88ff-jpjvs"]
Sep 17 17:35:43.199249 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.199214    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"]
Sep 17 17:35:43.199403 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.199328    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.213244 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.213212    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:43.225898 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.225879    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v5hwp\""
Sep 17 17:35:43.227179 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.227155    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 17:35:43.227263 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.227166    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 17:35:43.231958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.231941    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 17:35:43.233956 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.233940    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 17:35:43.235980 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.235963    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l58cx\""
Sep 17 17:35:43.251709 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.251686    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"]
Sep 17 17:35:43.253574 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.253523    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 17:35:43.267836 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.267811    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qzvww"]
Sep 17 17:35:43.286022 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.286001    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.289766 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.289747    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"]
Sep 17 17:35:43.289867 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.289849    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:35:43.290021 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.289850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:35:43.290097 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.290033    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:35:43.302791 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.302773    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:43.303031 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.303012    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:35:43.303173 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.303149    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2pzr6\""
Sep 17 17:35:43.303278 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.303217    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:35:43.303278 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.303234    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:35:43.303527 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.303510    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:43.303658 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.303637    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:43.305804 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.305784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-certificates\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.305903 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.305824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:43.305903 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.305850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbh2\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-kube-api-access-5rbh2\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.305903 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.305879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:43.306060 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.305958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-image-registry-private-configuration\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.306060 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.305991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-trusted-ca\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.306060 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.306011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-bound-sa-token\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.306060 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.306052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-installation-pull-secrets\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.306221 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.306114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.306221 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.306137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbee130d-e0e8-40c2-a388-297fc13cb5b3-ca-trust-extracted\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.308317 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.308299    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:43.311148 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.311128    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rcl8d"]
Sep 17 17:35:43.311253 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.311240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.335304 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.335286    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"]
Sep 17 17:35:43.335426 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.335411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.339593 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.339563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w6xmz\""
Sep 17 17:35:43.339724 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.339696    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wr2c\""
Sep 17 17:35:43.339800 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.339631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wffz7\""
Sep 17 17:35:43.340991 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.340971    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 17:35:43.343067 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.343043    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:35:43.346364 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.346344    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:35:43.346455 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.346437    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:35:43.346673 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.346652    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:35:43.355514 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.355482    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:35:43.355609 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.355533    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:35:43.357057 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.357038    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksc59\""
Sep 17 17:35:43.357536 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.357520    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"]
Sep 17 17:35:43.357664 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.357648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.364128 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.364107    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:35:43.379438 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.379414    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"]
Sep 17 17:35:43.379536 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.379446    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-545f7f88ff-jpjvs"]
Sep 17 17:35:43.379536 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.379472    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qzvww"]
Sep 17 17:35:43.379536 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.379491    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"]
Sep 17 17:35:43.379721 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.379536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.383246 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.383221    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:35:43.384685 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.384667    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:35:43.388424 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.388404    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcl8d"]
Sep 17 17:35:43.406956 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.406932    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:43.407068 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.406977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112f5254-705f-49a9-b374-1146b06e3279-cert\") pod \"ingress-canary-qzvww\" (UID: \"112f5254-705f-49a9-b374-1146b06e3279\") " pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.407068 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407007    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8bmc\" (UniqueName: \"kubernetes.io/projected/112f5254-705f-49a9-b374-1146b06e3279-kube-api-access-x8bmc\") pod \"ingress-canary-qzvww\" (UID: \"112f5254-705f-49a9-b374-1146b06e3279\") " pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.407068 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407039    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-image-registry-private-configuration\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407231 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-trusted-ca\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407231 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-bound-sa-token\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407231 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-installation-pull-secrets\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407231 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407412 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407232    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbee130d-e0e8-40c2-a388-297fc13cb5b3-ca-trust-extracted\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407412 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407267    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/189779ff-0e0a-4faf-9a17-9dd79c5856b1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb\" (UID: \"189779ff-0e0a-4faf-9a17-9dd79c5856b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.407412 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6szgn\" (UniqueName: \"kubernetes.io/projected/189779ff-0e0a-4faf-9a17-9dd79c5856b1-kube-api-access-6szgn\") pod \"managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb\" (UID: \"189779ff-0e0a-4faf-9a17-9dd79c5856b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.407412 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-certificates\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407412 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:43.407636 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407439    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbh2\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-kube-api-access-5rbh2\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.407636 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.407595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:43.407743 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.407698    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:43.407743 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.407715    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:35:43.407842 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.407773    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:43.907760813 +0000 UTC m=+35.095196640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:35:43.408111 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.408018    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:43.408111 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.408085    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:43.908067564 +0000 UTC m=+35.095503413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:43.408380 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.408349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-certificates\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.408680 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.408658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-trusted-ca\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.408755 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.408724    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbee130d-e0e8-40c2-a388-297fc13cb5b3-ca-trust-extracted\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.411987 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.411686    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:35:43.411987 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.411686    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:35:43.412711 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.412684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-installation-pull-secrets\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.413252 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.413203    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"]
Sep 17 17:35:43.413610 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.413573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-image-registry-private-configuration\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.426174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.426152    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-bound-sa-token\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.429819 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.429795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbh2\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-kube-api-access-5rbh2\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.508702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508640    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-ca\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.508702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508667    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.508887 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gxg\" (UniqueName: \"kubernetes.io/projected/918b1aa2-127d-4059-86e0-e1f8730b4ff3-kube-api-access-t4gxg\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.508887 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508783    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112f5254-705f-49a9-b374-1146b06e3279-cert\") pod \"ingress-canary-qzvww\" (UID: \"112f5254-705f-49a9-b374-1146b06e3279\") " pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.508887 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43537637-3dfa-41b6-a106-2963abd90828-tmp-dir\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.508887 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8bmc\" (UniqueName: \"kubernetes.io/projected/112f5254-705f-49a9-b374-1146b06e3279-kube-api-access-x8bmc\") pod \"ingress-canary-qzvww\" (UID: \"112f5254-705f-49a9-b374-1146b06e3279\") " pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.509081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508891    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.509081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b2d69-e3e0-4a79-9a37-b64477340405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.509081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6v4x\" (UniqueName: \"kubernetes.io/projected/dd9b2d69-e3e0-4a79-9a37-b64477340405-kube-api-access-z6v4x\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.509081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508975    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/918b1aa2-127d-4059-86e0-e1f8730b4ff3-tmp\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.509081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.508994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-hub\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.509081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.509067    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6n8\" (UniqueName: \"kubernetes.io/projected/43537637-3dfa-41b6-a106-2963abd90828-kube-api-access-hn6n8\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.509311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.509104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/918b1aa2-127d-4059-86e0-e1f8730b4ff3-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.509311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.509172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.509311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.509194    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43537637-3dfa-41b6-a106-2963abd90828-config-volume\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.509311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.509216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/189779ff-0e0a-4faf-9a17-9dd79c5856b1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb\" (UID: \"189779ff-0e0a-4faf-9a17-9dd79c5856b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.509311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.509233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6szgn\" (UniqueName: \"kubernetes.io/projected/189779ff-0e0a-4faf-9a17-9dd79c5856b1-kube-api-access-6szgn\") pod \"managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb\" (UID: \"189779ff-0e0a-4faf-9a17-9dd79c5856b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.511212 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.511181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/112f5254-705f-49a9-b374-1146b06e3279-cert\") pod \"ingress-canary-qzvww\" (UID: \"112f5254-705f-49a9-b374-1146b06e3279\") " pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.511401 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.511385    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/189779ff-0e0a-4faf-9a17-9dd79c5856b1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb\" (UID: \"189779ff-0e0a-4faf-9a17-9dd79c5856b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.522735 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.522711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6szgn\" (UniqueName: \"kubernetes.io/projected/189779ff-0e0a-4faf-9a17-9dd79c5856b1-kube-api-access-6szgn\") pod \"managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb\" (UID: \"189779ff-0e0a-4faf-9a17-9dd79c5856b1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.522814 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.522732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8bmc\" (UniqueName: \"kubernetes.io/projected/112f5254-705f-49a9-b374-1146b06e3279-kube-api-access-x8bmc\") pod \"ingress-canary-qzvww\" (UID: \"112f5254-705f-49a9-b374-1146b06e3279\") " pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.597184 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.597161    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qzvww"
Sep 17 17:35:43.609524 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609506    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.609623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609531    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43537637-3dfa-41b6-a106-2963abd90828-config-volume\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.609623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-ca\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.609623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.609623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gxg\" (UniqueName: \"kubernetes.io/projected/918b1aa2-127d-4059-86e0-e1f8730b4ff3-kube-api-access-t4gxg\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.609786 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43537637-3dfa-41b6-a106-2963abd90828-tmp-dir\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.609786 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609677    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.609786 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.609693    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:43.609786 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b2d69-e3e0-4a79-9a37-b64477340405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.609786 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6v4x\" (UniqueName: \"kubernetes.io/projected/dd9b2d69-e3e0-4a79-9a37-b64477340405-kube-api-access-z6v4x\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.609786 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.609763    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:44.109744592 +0000 UTC m=+35.297180435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:43.610123 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/918b1aa2-127d-4059-86e0-e1f8730b4ff3-tmp\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.610123 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-hub\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.610123 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6n8\" (UniqueName: \"kubernetes.io/projected/43537637-3dfa-41b6-a106-2963abd90828-kube-api-access-hn6n8\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.610123 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/918b1aa2-127d-4059-86e0-e1f8730b4ff3-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.610123 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.609983    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43537637-3dfa-41b6-a106-2963abd90828-tmp-dir\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.610123 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.610104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43537637-3dfa-41b6-a106-2963abd90828-config-volume\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.610422 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.610229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/918b1aa2-127d-4059-86e0-e1f8730b4ff3-tmp\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.610671 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.610647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd9b2d69-e3e0-4a79-9a37-b64477340405-additional-service-ca\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.612201 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.612176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.612314 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.612294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.612407 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.612387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-ca\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.612492 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.612475    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dd9b2d69-e3e0-4a79-9a37-b64477340405-hub\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.612655 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.612636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/918b1aa2-127d-4059-86e0-e1f8730b4ff3-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.630975 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.630953    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:35:43.631296 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.631268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6n8\" (UniqueName: \"kubernetes.io/projected/43537637-3dfa-41b6-a106-2963abd90828-kube-api-access-hn6n8\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:43.632159 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.632140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gxg\" (UniqueName: \"kubernetes.io/projected/918b1aa2-127d-4059-86e0-e1f8730b4ff3-kube-api-access-t4gxg\") pod \"klusterlet-addon-workmgr-6d8bc94c8f-chk4k\" (UID: \"918b1aa2-127d-4059-86e0-e1f8730b4ff3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.632241 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.632222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6v4x\" (UniqueName: \"kubernetes.io/projected/dd9b2d69-e3e0-4a79-9a37-b64477340405-kube-api-access-z6v4x\") pod \"cluster-proxy-proxy-agent-764d9cd55d-r59vt\" (UID: \"dd9b2d69-e3e0-4a79-9a37-b64477340405\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.668289 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.668268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:43.689843 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.689817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:35:43.912213 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.911564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:43.912213 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.911638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:43.912213 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.911763    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:43.912213 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.911772    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:43.912213 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.911791    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:35:43.912213 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.911823    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:44.911805179 +0000 UTC m=+36.099241009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:43.912213 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:43.911849    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:44.911829308 +0000 UTC m=+36.099265149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:35:43.933521 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.933481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qzvww"]
Sep 17 17:35:43.936142 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.936026    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"]
Sep 17 17:35:43.939025 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:43.938633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112f5254_705f_49a9_b374_1146b06e3279.slice/crio-04f3458f487a6587a30b29916428713fb683aaf9c600404e8c5cc890260285ff WatchSource:0}: Error finding container 04f3458f487a6587a30b29916428713fb683aaf9c600404e8c5cc890260285ff: Status 404 returned error can't find the container with id 04f3458f487a6587a30b29916428713fb683aaf9c600404e8c5cc890260285ff
Sep 17 17:35:43.940628 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:43.940600    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod189779ff_0e0a_4faf_9a17_9dd79c5856b1.slice/crio-c1a47ae79283b994fe4cccb5c2277e61d947536846dcb73f855c3e2495e2c37b WatchSource:0}: Error finding container c1a47ae79283b994fe4cccb5c2277e61d947536846dcb73f855c3e2495e2c37b: Status 404 returned error can't find the container with id c1a47ae79283b994fe4cccb5c2277e61d947536846dcb73f855c3e2495e2c37b
Sep 17 17:35:43.962729 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:43.962702    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"]
Sep 17 17:35:43.970755 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:35:43.970716    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918b1aa2_127d_4059_86e0_e1f8730b4ff3.slice/crio-ba709cc3b66e935e6cc79b88835b959243c46db0fdaa86fcb870604b7acd7a86 WatchSource:0}: Error finding container ba709cc3b66e935e6cc79b88835b959243c46db0fdaa86fcb870604b7acd7a86: Status 404 returned error can't find the container with id ba709cc3b66e935e6cc79b88835b959243c46db0fdaa86fcb870604b7acd7a86
Sep 17 17:35:44.113562 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.113532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:44.114226 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:44.113577    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:44.114226 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:44.113632    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:45.113619532 +0000 UTC m=+36.301055358 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:44.163220 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.161171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"]
Sep 17 17:35:44.550612 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.550545    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" event={"ID":"dd9b2d69-e3e0-4a79-9a37-b64477340405","Type":"ContainerStarted","Data":"9becca99c9f78ace92ccc65551fe99de0f65f93d41c52d012141895c8047cfab"}
Sep 17 17:35:44.555531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.555496    2562 generic.go:356] "Generic (PLEG): container finished" podID="48fdef06-8806-4e8d-bc0c-794bdb713240" containerID="dcd894ad61062c58005341f543d55f1c2d13b65a8781b84af900b9c3b982d1e4" exitCode=0
Sep 17 17:35:44.555673 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.555581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerDied","Data":"dcd894ad61062c58005341f543d55f1c2d13b65a8781b84af900b9c3b982d1e4"}
Sep 17 17:35:44.560143 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.560057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k" event={"ID":"918b1aa2-127d-4059-86e0-e1f8730b4ff3","Type":"ContainerStarted","Data":"ba709cc3b66e935e6cc79b88835b959243c46db0fdaa86fcb870604b7acd7a86"}
Sep 17 17:35:44.564835 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.564797    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb" event={"ID":"189779ff-0e0a-4faf-9a17-9dd79c5856b1","Type":"ContainerStarted","Data":"c1a47ae79283b994fe4cccb5c2277e61d947536846dcb73f855c3e2495e2c37b"}
Sep 17 17:35:44.567125 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.567077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qzvww" event={"ID":"112f5254-705f-49a9-b374-1146b06e3279","Type":"ContainerStarted","Data":"04f3458f487a6587a30b29916428713fb683aaf9c600404e8c5cc890260285ff"}
Sep 17 17:35:44.920171 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.920064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:44.920402 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:44.920385    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:44.920469 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:44.920455    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:46.920437136 +0000 UTC m=+38.107872969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:44.920790 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:44.920774    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:44.920859 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:44.920794    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:35:44.920859 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:44.920837    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:46.920821575 +0000 UTC m=+38.108257402 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:35:44.920859 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:44.920260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:45.123405 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:45.123327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:45.123841 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:45.123639    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:45.123841 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:45.123703    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:47.123684686 +0000 UTC m=+38.311120518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:45.576576 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:45.576540    2562 generic.go:356] "Generic (PLEG): container finished" podID="48fdef06-8806-4e8d-bc0c-794bdb713240" containerID="f599adaf814092fbb2724fa23a010ea10ff731fb2378fce38f4255fc30697e39" exitCode=0
Sep 17 17:35:45.576756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:45.576602    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerDied","Data":"f599adaf814092fbb2724fa23a010ea10ff731fb2378fce38f4255fc30697e39"}
Sep 17 17:35:46.586249 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:46.585979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w52g2" event={"ID":"48fdef06-8806-4e8d-bc0c-794bdb713240","Type":"ContainerStarted","Data":"2cd277c8dbf892355cb26d6ba01814121b613fa619b4ef65c766b4f33a4b4804"}
Sep 17 17:35:46.617691 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:46.617504    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w52g2" podStartSLOduration=8.651299839 podStartE2EDuration="36.617481601s" totalImagesPullingTime="27.966181762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:46.614696303 +0000 UTC m=+37.802132153" watchObservedRunningTime="2026-09-17 17:35:46.617481601 +0000 UTC m=+37.804917451"
Sep 17 17:35:46.941886 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:46.941768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:46.941886 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:46.941827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:46.942133 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:46.941949    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:46.942133 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:46.941957    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:46.942133 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:46.941981    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:35:46.942133 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:46.942022    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:50.942001218 +0000 UTC m=+42.129437067 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:46.942133 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:46.942039    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:50.942032575 +0000 UTC m=+42.129468402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:35:47.143732 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:47.143697    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:47.144165 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:47.143964    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:47.144165 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:47.144029    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:51.144010789 +0000 UTC m=+42.331446620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:50.979198 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:50.979157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:50.979592 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:50.979275    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:50.979592 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:50.979337    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:50.979592 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:50.979400    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:50.979592 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:50.979411    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:35:50.979592 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:50.979411    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:58.979394144 +0000 UTC m=+50.166829975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:50.979592 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:50.979469    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:58.97945388 +0000 UTC m=+50.166889709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:35:51.180995 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:51.180957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:51.181178 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:51.181125    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:51.181246 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:51.181210    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:59.181188848 +0000 UTC m=+50.368624698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:53.604873 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.604842    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb" event={"ID":"189779ff-0e0a-4faf-9a17-9dd79c5856b1","Type":"ContainerStarted","Data":"a2022fc2a7ec02b55cdbc0a84ad9840aacde4507bee315fdb82b34a99c5b7ac6"}
Sep 17 17:35:53.606181 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.606156    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qzvww" event={"ID":"112f5254-705f-49a9-b374-1146b06e3279","Type":"ContainerStarted","Data":"f6532c88e4016190820381b06f04c15661c556c4d011e3e9a16f1054c72a7b57"}
Sep 17 17:35:53.608124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.608104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" event={"ID":"dd9b2d69-e3e0-4a79-9a37-b64477340405","Type":"ContainerStarted","Data":"25aa479e3ac0854bbe5873b032bb942e7fd0becbd4fe0f4f585b2a6b4449ddf6"}
Sep 17 17:35:53.608210 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.608135    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" event={"ID":"dd9b2d69-e3e0-4a79-9a37-b64477340405","Type":"ContainerStarted","Data":"0cbb8eba4de0c4841d46bfeb76153f466d3778772a8a73c7dfdf5bf59f14bb67"}
Sep 17 17:35:53.608210 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.608153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" event={"ID":"dd9b2d69-e3e0-4a79-9a37-b64477340405","Type":"ContainerStarted","Data":"208296ebab91f4011f42660bcfd160dd30c2a1a15a7fa6765d007040f9ebffc8"}
Sep 17 17:35:53.609443 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.609424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k" event={"ID":"918b1aa2-127d-4059-86e0-e1f8730b4ff3","Type":"ContainerStarted","Data":"17a2c4ceac21d53930b8c6bd7a76cacfd2d00aa3f59aecfe82c46b7b25b31d1c"}
Sep 17 17:35:53.609655 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.609638    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:53.611312 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.611294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:35:53.628264 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.628224    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb" podStartSLOduration=18.979874305 podStartE2EDuration="27.628210998s" totalImagesPullingTime="8.648336693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:53.627427169 +0000 UTC m=+44.814863020" watchObservedRunningTime="2026-09-17 17:35:53.628210998 +0000 UTC m=+44.815646847"
Sep 17 17:35:53.662674 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.662637    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qzvww" podStartSLOduration=2.01266247 podStartE2EDuration="10.662624767s" totalImagesPullingTime="8.649962297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:53.658148887 +0000 UTC m=+44.845584754" watchObservedRunningTime="2026-09-17 17:35:53.662624767 +0000 UTC m=+44.850060616"
Sep 17 17:35:53.695021 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.694968    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k" podStartSLOduration=19.075878954 podStartE2EDuration="27.694949476s" totalImagesPullingTime="8.619070522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:53.694234433 +0000 UTC m=+44.881670283" watchObservedRunningTime="2026-09-17 17:35:53.694949476 +0000 UTC m=+44.882385326"
Sep 17 17:35:53.726706 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:53.726650    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" podStartSLOduration=19.293622467 podStartE2EDuration="27.726628105s" totalImagesPullingTime="8.433005638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:53.725348067 +0000 UTC m=+44.912783916" watchObservedRunningTime="2026-09-17 17:35:53.726628105 +0000 UTC m=+44.914063955"
Sep 17 17:35:59.040804 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:59.040769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:35:59.041233 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:59.040838    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:35:59.041233 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:59.040949    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:35:59.041233 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:59.041027    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:15.04101101 +0000 UTC m=+66.228446837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:35:59.041233 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:59.040956    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:35:59.041233 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:59.041049    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:35:59.041233 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:59.041144    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:15.041130334 +0000 UTC m=+66.228566172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:35:59.242035 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:35:59.242005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:35:59.242181 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:59.242111    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:59.242181 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:35:59.242166    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:15.242154058 +0000 UTC m=+66.429589885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:08.541971 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:08.541885    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qkrtg"
Sep 17 17:36:15.062574 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.062526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:36:15.062574 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.062576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.062610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.062631    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.062646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.062733    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.062743    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.062741    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.062806    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:47.062792565 +0000 UTC m=+98.250228392 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:36:15.063168 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.062819    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:47.062813203 +0000 UTC m=+98.250249030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:36:15.065084 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.065069    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:36:15.065362 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.065344    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:36:15.065406 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.065361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:36:15.073511 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.073496    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:36:15.073571 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.073559    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:19.073541993 +0000 UTC m=+130.260977820 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : secret "metrics-daemon-secret" not found
Sep 17 17:36:15.075469 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.075447    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:36:15.076449 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.076431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8dfece24-349b-48df-84f2-5b78d5c7feb5-original-pull-secret\") pod \"global-pull-secret-syncer-n9dx8\" (UID: \"8dfece24-349b-48df-84f2-5b78d5c7feb5\") " pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:36:15.086025 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.086000    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqzpw\" (UniqueName: \"kubernetes.io/projected/9bba983c-c289-4cbe-a15f-5e3877032bdd-kube-api-access-xqzpw\") pod \"network-check-target-dk7v5\" (UID: \"9bba983c-c289-4cbe-a15f-5e3877032bdd\") " pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:36:15.103695 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.103674    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n9dx8"
Sep 17 17:36:15.114253 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.114235    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w6xmz\""
Sep 17 17:36:15.121784 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.121758    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:36:15.264385 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.264355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:36:15.264514 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.264496    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:15.264575 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:15.264554    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:47.264540711 +0000 UTC m=+98.451976538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:15.381324 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.381253    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n9dx8"]
Sep 17 17:36:15.384374 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:36:15.384347    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfece24_349b_48df_84f2_5b78d5c7feb5.slice/crio-6be065f3e10c17999b5b17745fa829bfdbde22f4bd44bb49d7dc5dd28968df6f WatchSource:0}: Error finding container 6be065f3e10c17999b5b17745fa829bfdbde22f4bd44bb49d7dc5dd28968df6f: Status 404 returned error can't find the container with id 6be065f3e10c17999b5b17745fa829bfdbde22f4bd44bb49d7dc5dd28968df6f
Sep 17 17:36:15.406902 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.406877    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dk7v5"]
Sep 17 17:36:15.409278 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:36:15.409259    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bba983c_c289_4cbe_a15f_5e3877032bdd.slice/crio-e2425bdcd035b851741e0475f9276eb6379e98b93b8da0d7da4d2f06adbba6d9 WatchSource:0}: Error finding container e2425bdcd035b851741e0475f9276eb6379e98b93b8da0d7da4d2f06adbba6d9: Status 404 returned error can't find the container with id e2425bdcd035b851741e0475f9276eb6379e98b93b8da0d7da4d2f06adbba6d9
Sep 17 17:36:15.675967 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.675890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dk7v5" event={"ID":"9bba983c-c289-4cbe-a15f-5e3877032bdd","Type":"ContainerStarted","Data":"e2425bdcd035b851741e0475f9276eb6379e98b93b8da0d7da4d2f06adbba6d9"}
Sep 17 17:36:15.676866 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:15.676847    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n9dx8" event={"ID":"8dfece24-349b-48df-84f2-5b78d5c7feb5","Type":"ContainerStarted","Data":"6be065f3e10c17999b5b17745fa829bfdbde22f4bd44bb49d7dc5dd28968df6f"}
Sep 17 17:36:20.695439 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:20.695403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n9dx8" event={"ID":"8dfece24-349b-48df-84f2-5b78d5c7feb5","Type":"ContainerStarted","Data":"5ec50fb9fa9b480a5f6d25ada09f164051f5544c9cf757725a1da61833b3bf62"}
Sep 17 17:36:20.696973 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:20.696947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dk7v5" event={"ID":"9bba983c-c289-4cbe-a15f-5e3877032bdd","Type":"ContainerStarted","Data":"f1043da2b85c60d8442e8c91cec38cd0d0908176321fd44eeaf666f2cd9abf3b"}
Sep 17 17:36:20.697086 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:20.696991    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:36:20.716980 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:20.716941    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n9dx8" podStartSLOduration=66.177071628 podStartE2EDuration="1m10.716906969s" totalImagesPullingTime="4.539835341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:20.716491043 +0000 UTC m=+71.903926894" watchObservedRunningTime="2026-09-17 17:36:20.716906969 +0000 UTC m=+71.904342818"
Sep 17 17:36:47.095511 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:47.095460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:36:47.095897 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:47.095623    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:36:47.095897 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:47.095637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:36:47.095897 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:47.095711    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:51.09569655 +0000 UTC m=+162.283132376 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:36:47.095897 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:47.095741    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 17:36:47.095897 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:47.095753    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-545f7f88ff-jpjvs: secret "image-registry-tls" not found
Sep 17 17:36:47.095897 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:47.095797    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls podName:bbee130d-e0e8-40c2-a388-297fc13cb5b3 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:51.095786341 +0000 UTC m=+162.283222174 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls") pod "image-registry-545f7f88ff-jpjvs" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3") : secret "image-registry-tls" not found
Sep 17 17:36:47.297837 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:47.297802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:36:47.298031 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:47.297949    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:47.298031 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:36:47.298001    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls podName:43537637-3dfa-41b6-a106-2963abd90828 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:51.297988514 +0000 UTC m=+162.485424341 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls") pod "dns-default-rcl8d" (UID: "43537637-3dfa-41b6-a106-2963abd90828") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:51.702766 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:51.702738    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dk7v5"
Sep 17 17:36:51.724029 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:36:51.723981    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dk7v5" podStartSLOduration=97.213052851 podStartE2EDuration="1m41.723958038s" totalImagesPullingTime="4.510905187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:36:20.73832622 +0000 UTC m=+71.925762070" watchObservedRunningTime="2026-09-17 17:36:51.723958038 +0000 UTC m=+102.911393887"
Sep 17 17:37:19.123566 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:19.123529    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:37:19.124131 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:19.123692    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:37:19.124131 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:19.123772    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs podName:60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6 nodeName:}" failed. No retries permitted until 2026-09-17 17:39:21.123752061 +0000 UTC m=+252.311187902 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs") pod "network-metrics-daemon-95wb9" (UID: "60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6") : secret "metrics-daemon-secret" not found
Sep 17 17:37:31.800348 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.800312    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pvmjd"]
Sep 17 17:37:31.803413 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.803399    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:31.806712 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.806690    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:37:31.806827 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.806697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:37:31.806901 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.806888    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vkst8\""
Sep 17 17:37:31.807339 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.807325    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:37:31.807415 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.807360    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:37:31.824008 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.823987    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pvmjd"]
Sep 17 17:37:31.912819 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.912790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-crio-socket\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:31.912984 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.912821    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8b8\" (UniqueName: \"kubernetes.io/projected/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-api-access-8f8b8\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:31.912984 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.912928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:31.912984 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.912963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:31.913088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:31.913006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-data-volume\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.013274 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.013245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-crio-socket\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.013274 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.013274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8b8\" (UniqueName: \"kubernetes.io/projected/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-api-access-8f8b8\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.013420 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.013317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.013420 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.013343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.013420 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.013359    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-crio-socket\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.013420 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.013387    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-data-volume\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.013540 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.013444    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:32.013540 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.013448    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.013540 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.013504    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:32.513487463 +0000 UTC m=+143.700923293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:32.013632 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.013541    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:32.513527107 +0000 UTC m=+143.700962934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.013747 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.013732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-data-volume\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.024254 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.024229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8b8\" (UniqueName: \"kubernetes.io/projected/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-api-access-8f8b8\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.517241 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.517210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.517241 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:32.517244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:32.517446 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.517316    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.517446 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.517339    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:32.517446 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.517366    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:33.517352787 +0000 UTC m=+144.704788614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:32.517446 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:32.517388    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:33.517376368 +0000 UTC m=+144.704812195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:33.523275 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:33.523246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:33.523275 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:33.523279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:33.523667 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:33.523356    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:33.523667 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:33.523387    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:33.523667 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:33.523406    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:35.523392469 +0000 UTC m=+146.710828296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:33.523667 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:33.523438    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:35.523425588 +0000 UTC m=+146.710861431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:33.951127 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:33.951053    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txcs7_7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f/dns-node-resolver/0.log"
Sep 17 17:37:34.950088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:34.950064    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4zxd_6badc097-8f99-421a-a042-3e1991f71fc0/node-ca/0.log"
Sep 17 17:37:35.539761 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:35.539716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:35.539761 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:35.539767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:35.540055 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:35.539853    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:35.540055 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:35.539855    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:35.540055 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:35.539940    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:39.539899324 +0000 UTC m=+150.727335156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:35.540055 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:35.540000    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:39.539988037 +0000 UTC m=+150.727423864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:35.550709 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:35.550682    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qzvww_112f5254-705f-49a9-b374-1146b06e3279/serve-healthcheck-canary/0.log"
Sep 17 17:37:39.568889 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:39.568860    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:39.568889 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:39.568891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:39.569304 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:39.568975    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:39.569304 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:39.569020    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:47.569007056 +0000 UTC m=+158.756442884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:39.569304 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:39.569103    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:39.569304 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:39.569153    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:37:47.569141856 +0000 UTC m=+158.756577683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:46.210019 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:46.209958    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" podUID="bbee130d-e0e8-40c2-a388-297fc13cb5b3"
Sep 17 17:37:46.222328 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:46.222295    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6" podUID="262e63c2-cf2a-40a8-b6d0-ded6aa3978f7"
Sep 17 17:37:46.319178 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:46.319135    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-95wb9" podUID="60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6"
Sep 17 17:37:46.349954 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:46.349902    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-rcl8d" podUID="43537637-3dfa-41b6-a106-2963abd90828"
Sep 17 17:37:46.940254 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:46.940226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:37:46.940417 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:46.940226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:37:46.940417 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:46.940226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:37:47.629366 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:47.629326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:47.629366 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:47.629365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:47.629770 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:47.629483    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:47.629770 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:47.629556    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls podName:3f1fcef5-5357-408f-ae74-8e7b6ae4435b nodeName:}" failed. No retries permitted until 2026-09-17 17:38:03.629541018 +0000 UTC m=+174.816976845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pvmjd" (UID: "3f1fcef5-5357-408f-ae74-8e7b6ae4435b") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:47.629844 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:47.629835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:37:51.157057 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.157010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:37:51.157553 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.157072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:37:51.157553 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:51.157208    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 17:37:51.157553 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:37:51.157299    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert podName:262e63c2-cf2a-40a8-b6d0-ded6aa3978f7 nodeName:}" failed. No retries permitted until 2026-09-17 17:39:53.157275598 +0000 UTC m=+284.344711425 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-hkrf6" (UID: "262e63c2-cf2a-40a8-b6d0-ded6aa3978f7") : secret "networking-console-plugin-cert" not found
Sep 17 17:37:51.159464 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.159439    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"image-registry-545f7f88ff-jpjvs\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") " pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:37:51.357755 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.357726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:37:51.359973 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.359944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43537637-3dfa-41b6-a106-2963abd90828-metrics-tls\") pod \"dns-default-rcl8d\" (UID: \"43537637-3dfa-41b6-a106-2963abd90828\") " pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:37:51.444582 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.444518    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksc59\""
Sep 17 17:37:51.444582 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.444525    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l58cx\""
Sep 17 17:37:51.451697 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.451672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:37:51.451697 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.451689    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:37:51.737582 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.737553    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rcl8d"]
Sep 17 17:37:51.740694 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.740672    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-545f7f88ff-jpjvs"]
Sep 17 17:37:51.740967 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:37:51.740939    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43537637_3dfa_41b6_a106_2963abd90828.slice/crio-6ee5b5dbc64a75bb133a6848f08bb05d579a9316fc224b4d58df64165a913cc2 WatchSource:0}: Error finding container 6ee5b5dbc64a75bb133a6848f08bb05d579a9316fc224b4d58df64165a913cc2: Status 404 returned error can't find the container with id 6ee5b5dbc64a75bb133a6848f08bb05d579a9316fc224b4d58df64165a913cc2
Sep 17 17:37:51.743614 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:37:51.743549    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbee130d_e0e8_40c2_a388_297fc13cb5b3.slice/crio-5ad531b207d1f61f3b42a952fac0d6c668a0f65744da213209a14e4a663f0dff WatchSource:0}: Error finding container 5ad531b207d1f61f3b42a952fac0d6c668a0f65744da213209a14e4a663f0dff: Status 404 returned error can't find the container with id 5ad531b207d1f61f3b42a952fac0d6c668a0f65744da213209a14e4a663f0dff
Sep 17 17:37:51.955717 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.955678    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" event={"ID":"bbee130d-e0e8-40c2-a388-297fc13cb5b3","Type":"ContainerStarted","Data":"b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c"}
Sep 17 17:37:51.955899 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.955738    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" event={"ID":"bbee130d-e0e8-40c2-a388-297fc13cb5b3","Type":"ContainerStarted","Data":"5ad531b207d1f61f3b42a952fac0d6c668a0f65744da213209a14e4a663f0dff"}
Sep 17 17:37:51.955899 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.955792    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:37:51.956828 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.956807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcl8d" event={"ID":"43537637-3dfa-41b6-a106-2963abd90828","Type":"ContainerStarted","Data":"6ee5b5dbc64a75bb133a6848f08bb05d579a9316fc224b4d58df64165a913cc2"}
Sep 17 17:37:51.985713 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:51.985666    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" podStartSLOduration=161.985652355 podStartE2EDuration="2m41.985652355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:51.984629603 +0000 UTC m=+163.172065451" watchObservedRunningTime="2026-09-17 17:37:51.985652355 +0000 UTC m=+163.173088493"
Sep 17 17:37:52.961468 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:52.961435    2562 generic.go:356] "Generic (PLEG): container finished" podID="918b1aa2-127d-4059-86e0-e1f8730b4ff3" containerID="17a2c4ceac21d53930b8c6bd7a76cacfd2d00aa3f59aecfe82c46b7b25b31d1c" exitCode=1
Sep 17 17:37:52.961891 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:52.961504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k" event={"ID":"918b1aa2-127d-4059-86e0-e1f8730b4ff3","Type":"ContainerDied","Data":"17a2c4ceac21d53930b8c6bd7a76cacfd2d00aa3f59aecfe82c46b7b25b31d1c"}
Sep 17 17:37:52.961891 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:52.961881    2562 scope.go:117] "RemoveContainer" containerID="17a2c4ceac21d53930b8c6bd7a76cacfd2d00aa3f59aecfe82c46b7b25b31d1c"
Sep 17 17:37:52.963301 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:52.963273    2562 generic.go:356] "Generic (PLEG): container finished" podID="189779ff-0e0a-4faf-9a17-9dd79c5856b1" containerID="a2022fc2a7ec02b55cdbc0a84ad9840aacde4507bee315fdb82b34a99c5b7ac6" exitCode=255
Sep 17 17:37:52.963426 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:52.963351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb" event={"ID":"189779ff-0e0a-4faf-9a17-9dd79c5856b1","Type":"ContainerDied","Data":"a2022fc2a7ec02b55cdbc0a84ad9840aacde4507bee315fdb82b34a99c5b7ac6"}
Sep 17 17:37:52.963812 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:52.963792    2562 scope.go:117] "RemoveContainer" containerID="a2022fc2a7ec02b55cdbc0a84ad9840aacde4507bee315fdb82b34a99c5b7ac6"
Sep 17 17:37:53.610526 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.610460    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:37:53.631706 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.631681    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb"
Sep 17 17:37:53.668966 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.668950    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:37:53.970442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.970342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b6dfbff5d-8xdqb" event={"ID":"189779ff-0e0a-4faf-9a17-9dd79c5856b1","Type":"ContainerStarted","Data":"a27026f6af6383a2c418c9504f3bbd65c1d87aabc46ccc03436ace8ddaea17f3"}
Sep 17 17:37:53.971965 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.971934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k" event={"ID":"918b1aa2-127d-4059-86e0-e1f8730b4ff3","Type":"ContainerStarted","Data":"d5dfc9cdfc8fd75c69be19e99f6b44ffb848a4631d2675742d01fa9d6bf78a49"}
Sep 17 17:37:53.972103 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.972089    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:37:53.972626 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.972606    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8bc94c8f-chk4k"
Sep 17 17:37:53.973620 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.973600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcl8d" event={"ID":"43537637-3dfa-41b6-a106-2963abd90828","Type":"ContainerStarted","Data":"d6b5db787d2c835eade71f853768d85e658a83bebcea78592493ad844745fed4"}
Sep 17 17:37:53.973713 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.973640    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rcl8d" event={"ID":"43537637-3dfa-41b6-a106-2963abd90828","Type":"ContainerStarted","Data":"4990ce421b82697946f3ab391a962551c4df3ae4a9589d8c55eeb716d8e5347c"}
Sep 17 17:37:53.973769 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:53.973740    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:37:54.027466 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:54.027414    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rcl8d" podStartSLOduration=129.457326196 podStartE2EDuration="2m11.027402309s" totalImagesPullingTime="1.570076113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:54.008657283 +0000 UTC m=+165.196093133" watchObservedRunningTime="2026-09-17 17:37:54.027402309 +0000 UTC m=+165.214838154"
Sep 17 17:37:57.272051 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:37:57.272013    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:38:03.647833 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:03.647754    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:38:03.650124 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:03.650099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f1fcef5-5357-408f-ae74-8e7b6ae4435b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pvmjd\" (UID: \"3f1fcef5-5357-408f-ae74-8e7b6ae4435b\") " pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:38:03.912261 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:03.912169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pvmjd"
Sep 17 17:38:03.979401 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:03.979358    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rcl8d"
Sep 17 17:38:04.187676 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:04.187583    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pvmjd"]
Sep 17 17:38:04.189981 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:38:04.189954    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1fcef5_5357_408f_ae74_8e7b6ae4435b.slice/crio-c605a81053bb334dd41f1957d419326ce8d1876104d739e85707827ec965ba53 WatchSource:0}: Error finding container c605a81053bb334dd41f1957d419326ce8d1876104d739e85707827ec965ba53: Status 404 returned error can't find the container with id c605a81053bb334dd41f1957d419326ce8d1876104d739e85707827ec965ba53
Sep 17 17:38:05.012627 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:05.012596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvmjd" event={"ID":"3f1fcef5-5357-408f-ae74-8e7b6ae4435b","Type":"ContainerStarted","Data":"f63c8f644f63c942773ccca5ab6f162f91201f7ad0448146e8051f12f0a23c36"}
Sep 17 17:38:05.012956 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:05.012674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvmjd" event={"ID":"3f1fcef5-5357-408f-ae74-8e7b6ae4435b","Type":"ContainerStarted","Data":"2ce6ac37f2a78895c2466d5f590a5aa9e81d46d4c7977d250c0594ba89c0b100"}
Sep 17 17:38:05.012956 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:05.012693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvmjd" event={"ID":"3f1fcef5-5357-408f-ae74-8e7b6ae4435b","Type":"ContainerStarted","Data":"c605a81053bb334dd41f1957d419326ce8d1876104d739e85707827ec965ba53"}
Sep 17 17:38:07.020708 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:07.020672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pvmjd" event={"ID":"3f1fcef5-5357-408f-ae74-8e7b6ae4435b","Type":"ContainerStarted","Data":"4acd958e8ae65984d677257250262b4c8bf7d2efd9cd212bfee1030a83f7125c"}
Sep 17 17:38:11.456562 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:11.456525    2562 patch_prober.go:28] interesting pod/image-registry-545f7f88ff-jpjvs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 17:38:11.456946 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:11.456591    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" podUID="bbee130d-e0e8-40c2-a388-297fc13cb5b3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:38:12.900818 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.900763    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pvmjd" podStartSLOduration=40.047764627 podStartE2EDuration="41.900747103s" totalImagesPullingTime="1.852982476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:07.04473723 +0000 UTC m=+178.232173079" watchObservedRunningTime="2026-09-17 17:38:12.900747103 +0000 UTC m=+184.088182951"
Sep 17 17:38:12.901426 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.901409    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nbzqn"]
Sep 17 17:38:12.906606 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.906591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:12.909775 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.909751    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:38:12.909889 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.909826    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:38:12.910105 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.910090    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:38:12.910105 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.910099    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:38:12.910224 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.910101    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:38:12.910481 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.910467    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:38:12.933308 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.933291    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5v6jf\""
Sep 17 17:38:12.967392 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:12.967373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:38:13.017168 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017290 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-sys\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017290 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mhz\" (UniqueName: \"kubernetes.io/projected/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-kube-api-access-w4mhz\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017290 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-textfile\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017290 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-tls\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017290 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-metrics-client-ca\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-root\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-wtmp\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.017546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.017377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118226 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mhz\" (UniqueName: \"kubernetes.io/projected/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-kube-api-access-w4mhz\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118388 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118231    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-textfile\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118388 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-tls\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118388 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-metrics-client-ca\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118388 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-root\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118604 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-root\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118657 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-textfile\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118826 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-wtmp\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.118989 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-wtmp\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.119137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.118965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-metrics-client-ca\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.119233 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.119028    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.119233 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.119223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.119357 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.119271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-sys\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.119412 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.119361    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-sys\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.119841 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.119820    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-accelerators-collector-config\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.121266 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.121245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.121658 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.121641    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-node-exporter-tls\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.128395 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.128373    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mhz\" (UniqueName: \"kubernetes.io/projected/3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac-kube-api-access-w4mhz\") pod \"node-exporter-nbzqn\" (UID: \"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac\") " pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.215059 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:13.215003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nbzqn"
Sep 17 17:38:13.226649 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:38:13.226627    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3efc75_ec3b_4b78_a76d_bc9c9d4c0cac.slice/crio-7cfb40f3f29d59b9e03d46e159bb7532d53a827808073e8bcbe06c994765b4d1 WatchSource:0}: Error finding container 7cfb40f3f29d59b9e03d46e159bb7532d53a827808073e8bcbe06c994765b4d1: Status 404 returned error can't find the container with id 7cfb40f3f29d59b9e03d46e159bb7532d53a827808073e8bcbe06c994765b4d1
Sep 17 17:38:14.045292 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:14.045254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbzqn" event={"ID":"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac","Type":"ContainerStarted","Data":"7cfb40f3f29d59b9e03d46e159bb7532d53a827808073e8bcbe06c994765b4d1"}
Sep 17 17:38:15.050304 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:15.050273    2562 generic.go:356] "Generic (PLEG): container finished" podID="3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac" containerID="3452636c2635804ee5cbe758bd25dfb2db916ddf8800b095932ad24698a526e1" exitCode=0
Sep 17 17:38:15.050676 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:15.050328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbzqn" event={"ID":"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac","Type":"ContainerDied","Data":"3452636c2635804ee5cbe758bd25dfb2db916ddf8800b095932ad24698a526e1"}
Sep 17 17:38:16.057649 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:16.057616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbzqn" event={"ID":"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac","Type":"ContainerStarted","Data":"367cc1a58e60772568917b5296a3ff5ed0cd8f7b26b2a1b4014fe2a0365030f3"}
Sep 17 17:38:16.058020 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:16.057667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nbzqn" event={"ID":"3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac","Type":"ContainerStarted","Data":"58eb14ea6dd533d576574e5ca54ad65938d440a6a82658f77c962da648971c71"}
Sep 17 17:38:16.083819 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:16.083776    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nbzqn" podStartSLOduration=3.100347398 podStartE2EDuration="4.083762754s" totalImagesPullingTime="923.791174ms" totalInitContainerRuntime="59.624182ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:16.083181534 +0000 UTC m=+187.270617394" watchObservedRunningTime="2026-09-17 17:38:16.083762754 +0000 UTC m=+187.271198603"
Sep 17 17:38:27.602482 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:27.602448    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-545f7f88ff-jpjvs"]
Sep 17 17:38:37.684167 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:37.684131    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qzvww_112f5254-705f-49a9-b374-1146b06e3279/serve-healthcheck-canary/0.log"
Sep 17 17:38:43.690785 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:43.690746    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" podUID="dd9b2d69-e3e0-4a79-9a37-b64477340405" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:38:52.620810 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.620747    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" podUID="bbee130d-e0e8-40c2-a388-297fc13cb5b3" containerName="registry" containerID="cri-o://b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c" gracePeriod=30
Sep 17 17:38:52.861370 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.861349    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:38:52.925924 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.925831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-certificates\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.925924 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.925901    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-installation-pull-secrets\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.926137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.925996    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-image-registry-private-configuration\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.926137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.926041    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-trusted-ca\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.926137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.926065    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.926137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.926119    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rbh2\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-kube-api-access-5rbh2\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.926346 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.926144    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbee130d-e0e8-40c2-a388-297fc13cb5b3-ca-trust-extracted\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.926346 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.926167    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-bound-sa-token\") pod \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\" (UID: \"bbee130d-e0e8-40c2-a388-297fc13cb5b3\") "
Sep 17 17:38:52.926444 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.926400    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:52.926500 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.926482    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:38:52.928839 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.928802    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:52.928990 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.928922    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-kube-api-access-5rbh2" (OuterVolumeSpecName: "kube-api-access-5rbh2") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "kube-api-access-5rbh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:52.928990 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.928942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:38:52.928990 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.928943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:52.929122 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.929004    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:38:52.935146 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:52.935123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee130d-e0e8-40c2-a388-297fc13cb5b3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bbee130d-e0e8-40c2-a388-297fc13cb5b3" (UID: "bbee130d-e0e8-40c2-a388-297fc13cb5b3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:38:53.026726 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026689    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-installation-pull-secrets\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.026726 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026721    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bbee130d-e0e8-40c2-a388-297fc13cb5b3-image-registry-private-configuration\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.026904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026733    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-trusted-ca\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.026904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026745    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.026904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026756    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rbh2\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-kube-api-access-5rbh2\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.026904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026767    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bbee130d-e0e8-40c2-a388-297fc13cb5b3-ca-trust-extracted\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.026904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026778    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bbee130d-e0e8-40c2-a388-297fc13cb5b3-bound-sa-token\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.026904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.026790    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bbee130d-e0e8-40c2-a388-297fc13cb5b3-registry-certificates\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:38:53.175022 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.174975    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbee130d-e0e8-40c2-a388-297fc13cb5b3" containerID="b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c" exitCode=0
Sep 17 17:38:53.175159 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.175029    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs"
Sep 17 17:38:53.175159 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.175063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" event={"ID":"bbee130d-e0e8-40c2-a388-297fc13cb5b3","Type":"ContainerDied","Data":"b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c"}
Sep 17 17:38:53.175159 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.175116    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-545f7f88ff-jpjvs" event={"ID":"bbee130d-e0e8-40c2-a388-297fc13cb5b3","Type":"ContainerDied","Data":"5ad531b207d1f61f3b42a952fac0d6c668a0f65744da213209a14e4a663f0dff"}
Sep 17 17:38:53.175159 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.175136    2562 scope.go:117] "RemoveContainer" containerID="b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c"
Sep 17 17:38:53.183076 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.183057    2562 scope.go:117] "RemoveContainer" containerID="b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c"
Sep 17 17:38:53.183331 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:38:53.183310    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c\": container with ID starting with b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c not found: ID does not exist" containerID="b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c"
Sep 17 17:38:53.183393 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.183337    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c"} err="failed to get container status \"b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c\": rpc error: code = NotFound desc = could not find container \"b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c\": container with ID starting with b49925441d2e239c8a41167a436a926460158dd06a427cb2928926d7a380102c not found: ID does not exist"
Sep 17 17:38:53.195769 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.195750    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-545f7f88ff-jpjvs"]
Sep 17 17:38:53.199509 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.199490    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-545f7f88ff-jpjvs"]
Sep 17 17:38:53.275595 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.275566    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbee130d-e0e8-40c2-a388-297fc13cb5b3" path="/var/lib/kubelet/pods/bbee130d-e0e8-40c2-a388-297fc13cb5b3/volumes"
Sep 17 17:38:53.691855 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:38:53.691823    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" podUID="dd9b2d69-e3e0-4a79-9a37-b64477340405" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:39:03.691432 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:03.691389    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" podUID="dd9b2d69-e3e0-4a79-9a37-b64477340405" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:39:03.691936 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:03.691482    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt"
Sep 17 17:39:03.692097 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:03.692077    2562 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"25aa479e3ac0854bbe5873b032bb942e7fd0becbd4fe0f4f585b2a6b4449ddf6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 17:39:03.692164 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:03.692123    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" podUID="dd9b2d69-e3e0-4a79-9a37-b64477340405" containerName="service-proxy" containerID="cri-o://25aa479e3ac0854bbe5873b032bb942e7fd0becbd4fe0f4f585b2a6b4449ddf6" gracePeriod=30
Sep 17 17:39:04.212585 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:04.212551    2562 generic.go:356] "Generic (PLEG): container finished" podID="dd9b2d69-e3e0-4a79-9a37-b64477340405" containerID="25aa479e3ac0854bbe5873b032bb942e7fd0becbd4fe0f4f585b2a6b4449ddf6" exitCode=2
Sep 17 17:39:04.212739 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:04.212603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" event={"ID":"dd9b2d69-e3e0-4a79-9a37-b64477340405","Type":"ContainerDied","Data":"25aa479e3ac0854bbe5873b032bb942e7fd0becbd4fe0f4f585b2a6b4449ddf6"}
Sep 17 17:39:04.212739 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:04.212635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-764d9cd55d-r59vt" event={"ID":"dd9b2d69-e3e0-4a79-9a37-b64477340405","Type":"ContainerStarted","Data":"2b74e5674528603ca8d04c5cda3a77a787a47fae36f68e44977b4341fdb2f15a"}
Sep 17 17:39:21.126190 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:21.126142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:39:21.128456 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:21.128435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6-metrics-certs\") pod \"network-metrics-daemon-95wb9\" (UID: \"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6\") " pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:39:21.275150 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:21.275127    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wr2c\""
Sep 17 17:39:21.283152 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:21.283126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-95wb9"
Sep 17 17:39:21.553546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:21.553512    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-95wb9"]
Sep 17 17:39:21.556581 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:39:21.556556    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f7f9e4_46c8_4cf4_9cc7_5c8211c27ae6.slice/crio-eb9861694f1ec46c36c5b2af0e1374ca937e5f8768bc9f2ff5f5e9ad5c3a9dc0 WatchSource:0}: Error finding container eb9861694f1ec46c36c5b2af0e1374ca937e5f8768bc9f2ff5f5e9ad5c3a9dc0: Status 404 returned error can't find the container with id eb9861694f1ec46c36c5b2af0e1374ca937e5f8768bc9f2ff5f5e9ad5c3a9dc0
Sep 17 17:39:22.268735 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:22.268703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-95wb9" event={"ID":"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6","Type":"ContainerStarted","Data":"eb9861694f1ec46c36c5b2af0e1374ca937e5f8768bc9f2ff5f5e9ad5c3a9dc0"}
Sep 17 17:39:23.275503 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:23.275470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-95wb9" event={"ID":"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6","Type":"ContainerStarted","Data":"7502f5e369605f84e838e281c07b1ccf76e368a8cfdaabb655aa78b74264250c"}
Sep 17 17:39:23.275861 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:23.275521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-95wb9" event={"ID":"60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6","Type":"ContainerStarted","Data":"0ec3f4d230c1952bf34413b3e61a32ec3076f7cf6a0facc16633f3eb709b0b2c"}
Sep 17 17:39:23.291595 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:23.291551    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-95wb9" podStartSLOduration=252.445169986 podStartE2EDuration="4m13.291538823s" totalImagesPullingTime="846.368837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:23.290428513 +0000 UTC m=+254.477864363" watchObservedRunningTime="2026-09-17 17:39:23.291538823 +0000 UTC m=+254.478974671"
Sep 17 17:39:49.941361 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:39:49.941325    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6" podUID="262e63c2-cf2a-40a8-b6d0-ded6aa3978f7"
Sep 17 17:39:50.355745 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:50.355720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:39:53.253418 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:53.253382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:39:53.255722 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:53.255694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/262e63c2-cf2a-40a8-b6d0-ded6aa3978f7-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-hkrf6\" (UID: \"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:39:53.358335 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:53.358298    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-v5hwp\""
Sep 17 17:39:53.366519 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:53.366499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"
Sep 17 17:39:53.647964 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:53.647878    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6"]
Sep 17 17:39:53.651102 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:39:53.651062    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262e63c2_cf2a_40a8_b6d0_ded6aa3978f7.slice/crio-405d19b82a2bd15b0a81527873f5dbef6e85c4798532e00e684b57142f09b36f WatchSource:0}: Error finding container 405d19b82a2bd15b0a81527873f5dbef6e85c4798532e00e684b57142f09b36f: Status 404 returned error can't find the container with id 405d19b82a2bd15b0a81527873f5dbef6e85c4798532e00e684b57142f09b36f
Sep 17 17:39:54.369380 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:54.369332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6" event={"ID":"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7","Type":"ContainerStarted","Data":"405d19b82a2bd15b0a81527873f5dbef6e85c4798532e00e684b57142f09b36f"}
Sep 17 17:39:55.373553 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:55.373521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6" event={"ID":"262e63c2-cf2a-40a8-b6d0-ded6aa3978f7","Type":"ContainerStarted","Data":"bf4c8a8729dc792a46c12a0f2d15f265a83efad1ed836e70770d7cd5207a3332"}
Sep 17 17:39:55.392019 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:39:55.391972    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-hkrf6" podStartSLOduration=268.361687476 podStartE2EDuration="4m29.391957465s" totalImagesPullingTime="1.030269989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:55.391007171 +0000 UTC m=+286.578443044" watchObservedRunningTime="2026-09-17 17:39:55.391957465 +0000 UTC m=+286.579393314"
Sep 17 17:40:09.151438 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:40:09.151409    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:44:00.835388 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.835307    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-wm72f"]
Sep 17 17:44:00.835905 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.835684    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbee130d-e0e8-40c2-a388-297fc13cb5b3" containerName="registry"
Sep 17 17:44:00.835905 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.835701    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbee130d-e0e8-40c2-a388-297fc13cb5b3" containerName="registry"
Sep 17 17:44:00.835905 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.835765    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbee130d-e0e8-40c2-a388-297fc13cb5b3" containerName="registry"
Sep 17 17:44:00.838563 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.838542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:00.841029 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.841003    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:44:00.841029 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.841005    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:44:00.841760 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.841743    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-dhl2v\""
Sep 17 17:44:00.841760 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.841752    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:44:00.853319 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:00.853295    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-wm72f"]
Sep 17 17:44:01.000295 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.000263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert\") pod \"kserve-controller-manager-9b487cb79-wm72f\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") " pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:01.000446 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.000304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4dx\" (UniqueName: \"kubernetes.io/projected/2d3e813a-45ed-429d-b304-0c1749a94b01-kube-api-access-2n4dx\") pod \"kserve-controller-manager-9b487cb79-wm72f\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") " pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:01.100833 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.100753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert\") pod \"kserve-controller-manager-9b487cb79-wm72f\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") " pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:01.100833 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.100798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4dx\" (UniqueName: \"kubernetes.io/projected/2d3e813a-45ed-429d-b304-0c1749a94b01-kube-api-access-2n4dx\") pod \"kserve-controller-manager-9b487cb79-wm72f\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") " pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:01.101028 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:44:01.100903    2562 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 17:44:01.101028 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:44:01.100997    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert podName:2d3e813a-45ed-429d-b304-0c1749a94b01 nodeName:}" failed. No retries permitted until 2026-09-17 17:44:01.600980947 +0000 UTC m=+532.788416773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert") pod "kserve-controller-manager-9b487cb79-wm72f" (UID: "2d3e813a-45ed-429d-b304-0c1749a94b01") : secret "kserve-webhook-server-cert" not found
Sep 17 17:44:01.112229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.112199    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4dx\" (UniqueName: \"kubernetes.io/projected/2d3e813a-45ed-429d-b304-0c1749a94b01-kube-api-access-2n4dx\") pod \"kserve-controller-manager-9b487cb79-wm72f\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") " pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:01.604542 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.604489    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert\") pod \"kserve-controller-manager-9b487cb79-wm72f\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") " pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:01.606950 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.606927    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert\") pod \"kserve-controller-manager-9b487cb79-wm72f\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") " pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:01.748184 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:01.748144    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:02.017227 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:02.017200    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-wm72f"]
Sep 17 17:44:02.019583 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:44:02.019551    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3e813a_45ed_429d_b304_0c1749a94b01.slice/crio-8392a6831d219dc1d11b95d4c404185684395d0dc1121ba4c9a14b0aeffb9cfc WatchSource:0}: Error finding container 8392a6831d219dc1d11b95d4c404185684395d0dc1121ba4c9a14b0aeffb9cfc: Status 404 returned error can't find the container with id 8392a6831d219dc1d11b95d4c404185684395d0dc1121ba4c9a14b0aeffb9cfc
Sep 17 17:44:02.020738 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:02.020721    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:44:02.163056 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:02.163020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-wm72f" event={"ID":"2d3e813a-45ed-429d-b304-0c1749a94b01","Type":"ContainerStarted","Data":"8392a6831d219dc1d11b95d4c404185684395d0dc1121ba4c9a14b0aeffb9cfc"}
Sep 17 17:44:05.175259 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:05.175157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-wm72f" event={"ID":"2d3e813a-45ed-429d-b304-0c1749a94b01","Type":"ContainerStarted","Data":"51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6"}
Sep 17 17:44:05.175619 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:05.175273    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:05.190963 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:05.190890    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-9b487cb79-wm72f" podStartSLOduration=2.467962218 podStartE2EDuration="5.190872385s" totalImagesPullingTime="2.722910167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:05.190164704 +0000 UTC m=+536.377600553" watchObservedRunningTime="2026-09-17 17:44:05.190872385 +0000 UTC m=+536.378308233"
Sep 17 17:44:36.183540 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:36.183508    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:45.394449 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.394412    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-wm72f"]
Sep 17 17:44:45.394833 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.394626    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-9b487cb79-wm72f" podUID="2d3e813a-45ed-429d-b304-0c1749a94b01" containerName="manager" containerID="cri-o://51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6" gracePeriod=10
Sep 17 17:44:45.421369 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.421345    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-phl6l"]
Sep 17 17:44:45.424369 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.424355    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.435256 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.435231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-phl6l"]
Sep 17 17:44:45.503770 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.503742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdr8\" (UniqueName: \"kubernetes.io/projected/6c182504-61e0-42d0-8c08-03149103893e-kube-api-access-vzdr8\") pod \"kserve-controller-manager-9b487cb79-phl6l\" (UID: \"6c182504-61e0-42d0-8c08-03149103893e\") " pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.503933 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.503777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c182504-61e0-42d0-8c08-03149103893e-cert\") pod \"kserve-controller-manager-9b487cb79-phl6l\" (UID: \"6c182504-61e0-42d0-8c08-03149103893e\") " pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.604788 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.604757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdr8\" (UniqueName: \"kubernetes.io/projected/6c182504-61e0-42d0-8c08-03149103893e-kube-api-access-vzdr8\") pod \"kserve-controller-manager-9b487cb79-phl6l\" (UID: \"6c182504-61e0-42d0-8c08-03149103893e\") " pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.604987 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.604795    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c182504-61e0-42d0-8c08-03149103893e-cert\") pod \"kserve-controller-manager-9b487cb79-phl6l\" (UID: \"6c182504-61e0-42d0-8c08-03149103893e\") " pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.607041 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.607017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c182504-61e0-42d0-8c08-03149103893e-cert\") pod \"kserve-controller-manager-9b487cb79-phl6l\" (UID: \"6c182504-61e0-42d0-8c08-03149103893e\") " pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.614313 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.614289    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdr8\" (UniqueName: \"kubernetes.io/projected/6c182504-61e0-42d0-8c08-03149103893e-kube-api-access-vzdr8\") pod \"kserve-controller-manager-9b487cb79-phl6l\" (UID: \"6c182504-61e0-42d0-8c08-03149103893e\") " pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.631555 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.631537    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:45.705729 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.705656    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2n4dx\" (UniqueName: \"kubernetes.io/projected/2d3e813a-45ed-429d-b304-0c1749a94b01-kube-api-access-2n4dx\") pod \"2d3e813a-45ed-429d-b304-0c1749a94b01\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") "
Sep 17 17:44:45.705869 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.705736    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert\") pod \"2d3e813a-45ed-429d-b304-0c1749a94b01\" (UID: \"2d3e813a-45ed-429d-b304-0c1749a94b01\") "
Sep 17 17:44:45.707759 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.707734    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert" (OuterVolumeSpecName: "cert") pod "2d3e813a-45ed-429d-b304-0c1749a94b01" (UID: "2d3e813a-45ed-429d-b304-0c1749a94b01"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:44:45.707866 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.707755    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3e813a-45ed-429d-b304-0c1749a94b01-kube-api-access-2n4dx" (OuterVolumeSpecName: "kube-api-access-2n4dx") pod "2d3e813a-45ed-429d-b304-0c1749a94b01" (UID: "2d3e813a-45ed-429d-b304-0c1749a94b01"). InnerVolumeSpecName "kube-api-access-2n4dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:44:45.774100 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.774056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:45.806372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.806334    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4dx\" (UniqueName: \"kubernetes.io/projected/2d3e813a-45ed-429d-b304-0c1749a94b01-kube-api-access-2n4dx\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:44:45.806372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:45.806370    2562 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2d3e813a-45ed-429d-b304-0c1749a94b01-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:44:46.043043 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.043010    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-phl6l"]
Sep 17 17:44:46.047173 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:44:46.047132    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c182504_61e0_42d0_8c08_03149103893e.slice/crio-4c4b3bbe4dd189581ae2795c99e11b657d2bed7c92777463f217246145c50473 WatchSource:0}: Error finding container 4c4b3bbe4dd189581ae2795c99e11b657d2bed7c92777463f217246145c50473: Status 404 returned error can't find the container with id 4c4b3bbe4dd189581ae2795c99e11b657d2bed7c92777463f217246145c50473
Sep 17 17:44:46.309090 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.308992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-phl6l" event={"ID":"6c182504-61e0-42d0-8c08-03149103893e","Type":"ContainerStarted","Data":"4c4b3bbe4dd189581ae2795c99e11b657d2bed7c92777463f217246145c50473"}
Sep 17 17:44:46.310090 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.310072    2562 generic.go:356] "Generic (PLEG): container finished" podID="2d3e813a-45ed-429d-b304-0c1749a94b01" containerID="51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6" exitCode=0
Sep 17 17:44:46.310174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.310106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-wm72f" event={"ID":"2d3e813a-45ed-429d-b304-0c1749a94b01","Type":"ContainerDied","Data":"51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6"}
Sep 17 17:44:46.310174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.310132    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-wm72f" event={"ID":"2d3e813a-45ed-429d-b304-0c1749a94b01","Type":"ContainerDied","Data":"8392a6831d219dc1d11b95d4c404185684395d0dc1121ba4c9a14b0aeffb9cfc"}
Sep 17 17:44:46.310174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.310152    2562 scope.go:117] "RemoveContainer" containerID="51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6"
Sep 17 17:44:46.310322 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.310166    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-9b487cb79-wm72f"
Sep 17 17:44:46.318008 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.317986    2562 scope.go:117] "RemoveContainer" containerID="51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6"
Sep 17 17:44:46.318290 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:44:46.318264    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6\": container with ID starting with 51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6 not found: ID does not exist" containerID="51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6"
Sep 17 17:44:46.318375 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.318302    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6"} err="failed to get container status \"51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6\": rpc error: code = NotFound desc = could not find container \"51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6\": container with ID starting with 51a514eb5076c23f374bc3e6e739575cb8421e7a57a4fe33787a72f33a0c51b6 not found: ID does not exist"
Sep 17 17:44:46.332904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.332880    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-wm72f"]
Sep 17 17:44:46.340404 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:46.340385    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-9b487cb79-wm72f"]
Sep 17 17:44:47.275443 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:47.275400    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d3e813a-45ed-429d-b304-0c1749a94b01" path="/var/lib/kubelet/pods/2d3e813a-45ed-429d-b304-0c1749a94b01/volumes"
Sep 17 17:44:47.315125 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:47.315095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-9b487cb79-phl6l" event={"ID":"6c182504-61e0-42d0-8c08-03149103893e","Type":"ContainerStarted","Data":"c4db1f9bafb1e93b5e5b7a045c1d5c49bf36a3b005d494971bc5e89dc4382c9b"}
Sep 17 17:44:47.315257 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:47.315225    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:44:47.358272 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:44:47.358226    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-9b487cb79-phl6l" podStartSLOduration=1.896131585 podStartE2EDuration="2.358211985s" totalImagesPullingTime="462.0804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:44:47.357634075 +0000 UTC m=+578.545069923" watchObservedRunningTime="2026-09-17 17:44:47.358211985 +0000 UTC m=+578.545647833"
Sep 17 17:45:18.323601 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:18.323573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-9b487cb79-phl6l"
Sep 17 17:45:19.761002 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.760967    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-26zsj"]
Sep 17 17:45:19.761354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.761215    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3e813a-45ed-429d-b304-0c1749a94b01" containerName="manager"
Sep 17 17:45:19.761354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.761226    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3e813a-45ed-429d-b304-0c1749a94b01" containerName="manager"
Sep 17 17:45:19.761354 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.761276    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d3e813a-45ed-429d-b304-0c1749a94b01" containerName="manager"
Sep 17 17:45:19.765211 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.765191    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:19.767875 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.767855    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 17:45:19.768008 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.767875    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rjkgd\""
Sep 17 17:45:19.777179 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.777153    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-26zsj"]
Sep 17 17:45:19.837544 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.837507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5dd334-765d-4c84-833a-c533ddb0005a-tls-certs\") pod \"model-serving-api-7b954ffc95-26zsj\" (UID: \"eb5dd334-765d-4c84-833a-c533ddb0005a\") " pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:19.837544 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.837547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bgq\" (UniqueName: \"kubernetes.io/projected/eb5dd334-765d-4c84-833a-c533ddb0005a-kube-api-access-c8bgq\") pod \"model-serving-api-7b954ffc95-26zsj\" (UID: \"eb5dd334-765d-4c84-833a-c533ddb0005a\") " pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:19.938858 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.938827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5dd334-765d-4c84-833a-c533ddb0005a-tls-certs\") pod \"model-serving-api-7b954ffc95-26zsj\" (UID: \"eb5dd334-765d-4c84-833a-c533ddb0005a\") " pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:19.938858 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.938862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bgq\" (UniqueName: \"kubernetes.io/projected/eb5dd334-765d-4c84-833a-c533ddb0005a-kube-api-access-c8bgq\") pod \"model-serving-api-7b954ffc95-26zsj\" (UID: \"eb5dd334-765d-4c84-833a-c533ddb0005a\") " pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:19.939075 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:45:19.938983    2562 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 17:45:19.939075 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:45:19.939052    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb5dd334-765d-4c84-833a-c533ddb0005a-tls-certs podName:eb5dd334-765d-4c84-833a-c533ddb0005a nodeName:}" failed. No retries permitted until 2026-09-17 17:45:20.439036242 +0000 UTC m=+611.626472069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/eb5dd334-765d-4c84-833a-c533ddb0005a-tls-certs") pod "model-serving-api-7b954ffc95-26zsj" (UID: "eb5dd334-765d-4c84-833a-c533ddb0005a") : secret "model-serving-api-tls" not found
Sep 17 17:45:19.947783 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:19.947761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bgq\" (UniqueName: \"kubernetes.io/projected/eb5dd334-765d-4c84-833a-c533ddb0005a-kube-api-access-c8bgq\") pod \"model-serving-api-7b954ffc95-26zsj\" (UID: \"eb5dd334-765d-4c84-833a-c533ddb0005a\") " pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:20.443434 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:20.443400    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5dd334-765d-4c84-833a-c533ddb0005a-tls-certs\") pod \"model-serving-api-7b954ffc95-26zsj\" (UID: \"eb5dd334-765d-4c84-833a-c533ddb0005a\") " pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:20.445730 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:20.445713    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb5dd334-765d-4c84-833a-c533ddb0005a-tls-certs\") pod \"model-serving-api-7b954ffc95-26zsj\" (UID: \"eb5dd334-765d-4c84-833a-c533ddb0005a\") " pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:20.674684 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:20.674643    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:20.945838 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:20.945809    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-26zsj"]
Sep 17 17:45:20.949203 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:45:20.949177    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5dd334_765d_4c84_833a_c533ddb0005a.slice/crio-f04dd6f5947c337e80e180ae960e2bc579b89549baafd8b05c727ab4c20cf4e5 WatchSource:0}: Error finding container f04dd6f5947c337e80e180ae960e2bc579b89549baafd8b05c727ab4c20cf4e5: Status 404 returned error can't find the container with id f04dd6f5947c337e80e180ae960e2bc579b89549baafd8b05c727ab4c20cf4e5
Sep 17 17:45:21.428235 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:21.428187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-26zsj" event={"ID":"eb5dd334-765d-4c84-833a-c533ddb0005a","Type":"ContainerStarted","Data":"f04dd6f5947c337e80e180ae960e2bc579b89549baafd8b05c727ab4c20cf4e5"}
Sep 17 17:45:23.437036 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:23.436984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-26zsj" event={"ID":"eb5dd334-765d-4c84-833a-c533ddb0005a","Type":"ContainerStarted","Data":"8aa4a6d30f0390e63c6dc65358cf78980653a582391d9a95481f4477dd260b97"}
Sep 17 17:45:23.437458 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:23.437121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:45:23.454385 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:23.454336    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-26zsj" podStartSLOduration=2.423528263 podStartE2EDuration="4.454321856s" totalImagesPullingTime="2.030793593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:23.453285276 +0000 UTC m=+614.640721126" watchObservedRunningTime="2026-09-17 17:45:23.454321856 +0000 UTC m=+614.641757704"
Sep 17 17:45:34.444501 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:45:34.444423    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-26zsj"
Sep 17 17:46:55.477601 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.477566    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"]
Sep 17 17:46:55.480372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.480356    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:55.482998 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.482972    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 17:46:55.483107 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.483085    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-m9dzw\""
Sep 17 17:46:55.491657 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.491633    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"]
Sep 17 17:46:55.578493 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.578456    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29de9135-b33d-465e-b192-946bde56ea41-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-62wp6\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:55.578657 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.578503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48k75\" (UniqueName: \"kubernetes.io/projected/29de9135-b33d-465e-b192-946bde56ea41-kube-api-access-48k75\") pod \"seaweedfs-tls-custom-6644bf5d6c-62wp6\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:55.679164 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.679115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29de9135-b33d-465e-b192-946bde56ea41-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-62wp6\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:55.679295 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.679178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48k75\" (UniqueName: \"kubernetes.io/projected/29de9135-b33d-465e-b192-946bde56ea41-kube-api-access-48k75\") pod \"seaweedfs-tls-custom-6644bf5d6c-62wp6\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:55.679511 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.679494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29de9135-b33d-465e-b192-946bde56ea41-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-62wp6\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:55.687236 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.687210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48k75\" (UniqueName: \"kubernetes.io/projected/29de9135-b33d-465e-b192-946bde56ea41-kube-api-access-48k75\") pod \"seaweedfs-tls-custom-6644bf5d6c-62wp6\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:55.789492 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:55.789466    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:46:56.060342 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:56.060266    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"]
Sep 17 17:46:56.063565 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:46:56.063530    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29de9135_b33d_465e_b192_946bde56ea41.slice/crio-fca7afa9e757fa49a360e9fac3564f0da2e4c4d91e0aff177fe045bf59c58bd1 WatchSource:0}: Error finding container fca7afa9e757fa49a360e9fac3564f0da2e4c4d91e0aff177fe045bf59c58bd1: Status 404 returned error can't find the container with id fca7afa9e757fa49a360e9fac3564f0da2e4c4d91e0aff177fe045bf59c58bd1
Sep 17 17:46:56.741597 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:56.741550    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6" event={"ID":"29de9135-b33d-465e-b192-946bde56ea41","Type":"ContainerStarted","Data":"fca7afa9e757fa49a360e9fac3564f0da2e4c4d91e0aff177fe045bf59c58bd1"}
Sep 17 17:46:58.750836 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:58.750802    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6" event={"ID":"29de9135-b33d-465e-b192-946bde56ea41","Type":"ContainerStarted","Data":"63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee"}
Sep 17 17:46:58.769157 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:58.769114    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6" podStartSLOduration=1.161084951 podStartE2EDuration="3.769098819s" totalImagesPullingTime="2.608013868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:58.767966561 +0000 UTC m=+709.955402410" watchObservedRunningTime="2026-09-17 17:46:58.769098819 +0000 UTC m=+709.956534667"
Sep 17 17:46:59.444423 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:46:59.444391    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"]
Sep 17 17:47:00.758051 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:00.757989    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6" podUID="29de9135-b33d-465e-b192-946bde56ea41" containerName="seaweedfs-tls-custom" containerID="cri-o://63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee" gracePeriod=30
Sep 17 17:47:01.989557 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:01.989535    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:47:02.024636 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.024571    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48k75\" (UniqueName: \"kubernetes.io/projected/29de9135-b33d-465e-b192-946bde56ea41-kube-api-access-48k75\") pod \"29de9135-b33d-465e-b192-946bde56ea41\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") "
Sep 17 17:47:02.024636 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.024624    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29de9135-b33d-465e-b192-946bde56ea41-data\") pod \"29de9135-b33d-465e-b192-946bde56ea41\" (UID: \"29de9135-b33d-465e-b192-946bde56ea41\") "
Sep 17 17:47:02.025922 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.025893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29de9135-b33d-465e-b192-946bde56ea41-data" (OuterVolumeSpecName: "data") pod "29de9135-b33d-465e-b192-946bde56ea41" (UID: "29de9135-b33d-465e-b192-946bde56ea41"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:47:02.026500 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.026479    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29de9135-b33d-465e-b192-946bde56ea41-kube-api-access-48k75" (OuterVolumeSpecName: "kube-api-access-48k75") pod "29de9135-b33d-465e-b192-946bde56ea41" (UID: "29de9135-b33d-465e-b192-946bde56ea41"). InnerVolumeSpecName "kube-api-access-48k75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:47:02.125886 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.125857    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48k75\" (UniqueName: \"kubernetes.io/projected/29de9135-b33d-465e-b192-946bde56ea41-kube-api-access-48k75\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:47:02.125886 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.125882    2562 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29de9135-b33d-465e-b192-946bde56ea41-data\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:47:02.767020 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.766978    2562 generic.go:356] "Generic (PLEG): container finished" podID="29de9135-b33d-465e-b192-946bde56ea41" containerID="63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee" exitCode=0
Sep 17 17:47:02.767349 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.767327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6" event={"ID":"29de9135-b33d-465e-b192-946bde56ea41","Type":"ContainerDied","Data":"63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee"}
Sep 17 17:47:02.767513 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.767499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6" event={"ID":"29de9135-b33d-465e-b192-946bde56ea41","Type":"ContainerDied","Data":"fca7afa9e757fa49a360e9fac3564f0da2e4c4d91e0aff177fe045bf59c58bd1"}
Sep 17 17:47:02.767652 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.767639    2562 scope.go:117] "RemoveContainer" containerID="63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee"
Sep 17 17:47:02.767949 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.767929    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"
Sep 17 17:47:02.780327 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.780307    2562 scope.go:117] "RemoveContainer" containerID="63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee"
Sep 17 17:47:02.780586 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:47:02.780568    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee\": container with ID starting with 63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee not found: ID does not exist" containerID="63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee"
Sep 17 17:47:02.780632 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.780595    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee"} err="failed to get container status \"63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee\": rpc error: code = NotFound desc = could not find container \"63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee\": container with ID starting with 63136915e3c33626035ad70ee023ec2d2a412b616814d1e9188fd64d47697fee not found: ID does not exist"
Sep 17 17:47:02.791272 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.791251    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"]
Sep 17 17:47:02.793443 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:02.793425    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-62wp6"]
Sep 17 17:47:03.276029 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:03.275996    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29de9135-b33d-465e-b192-946bde56ea41" path="/var/lib/kubelet/pods/29de9135-b33d-465e-b192-946bde56ea41/volumes"
Sep 17 17:47:16.439218 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.439186    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"]
Sep 17 17:47:16.439580 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.439435    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29de9135-b33d-465e-b192-946bde56ea41" containerName="seaweedfs-tls-custom"
Sep 17 17:47:16.439580 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.439446    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="29de9135-b33d-465e-b192-946bde56ea41" containerName="seaweedfs-tls-custom"
Sep 17 17:47:16.439580 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.439487    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="29de9135-b33d-465e-b192-946bde56ea41" containerName="seaweedfs-tls-custom"
Sep 17 17:47:16.443581 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.443562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.446008 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.445983    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 17:47:16.446173 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.446154    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-m9dzw\""
Sep 17 17:47:16.446547 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.446531    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 17 17:47:16.451415 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.451393    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"]
Sep 17 17:47:16.520584 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.520554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a55aa06e-fcc9-430a-af47-a148b7049736-data\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.520736 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.520610    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf2td\" (UniqueName: \"kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-kube-api-access-hf2td\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.520736 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.520708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.621426 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.621397    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.621583 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.621435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a55aa06e-fcc9-430a-af47-a148b7049736-data\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.621583 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.621465    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf2td\" (UniqueName: \"kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-kube-api-access-hf2td\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.621583 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:47:16.621542    2562 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Sep 17 17:47:16.621583 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:47:16.621562    2562 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2: secret "seaweedfs-tls-serving" not found
Sep 17 17:47:16.621736 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:47:16.621626    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-seaweedfs-tls-serving podName:a55aa06e-fcc9-430a-af47-a148b7049736 nodeName:}" failed. No retries permitted until 2026-09-17 17:47:17.121611329 +0000 UTC m=+728.309047156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-t7rt2" (UID: "a55aa06e-fcc9-430a-af47-a148b7049736") : secret "seaweedfs-tls-serving" not found
Sep 17 17:47:16.621857 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.621838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a55aa06e-fcc9-430a-af47-a148b7049736-data\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:16.632362 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:16.632331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf2td\" (UniqueName: \"kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-kube-api-access-hf2td\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:17.125235 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:17.125200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:17.127582 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:17.127548    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a55aa06e-fcc9-430a-af47-a148b7049736-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-t7rt2\" (UID: \"a55aa06e-fcc9-430a-af47-a148b7049736\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:17.352459 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:17.352428    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"
Sep 17 17:47:17.620014 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:17.619988    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2"]
Sep 17 17:47:17.622237 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:47:17.622210    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55aa06e_fcc9_430a_af47_a148b7049736.slice/crio-2e2e20c0f97ca03a106665e7d60e6e2e8d828c7f86c3b8ad31b681171abc4407 WatchSource:0}: Error finding container 2e2e20c0f97ca03a106665e7d60e6e2e8d828c7f86c3b8ad31b681171abc4407: Status 404 returned error can't find the container with id 2e2e20c0f97ca03a106665e7d60e6e2e8d828c7f86c3b8ad31b681171abc4407
Sep 17 17:47:17.823303 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:17.823272    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2" event={"ID":"a55aa06e-fcc9-430a-af47-a148b7049736","Type":"ContainerStarted","Data":"2e2e20c0f97ca03a106665e7d60e6e2e8d828c7f86c3b8ad31b681171abc4407"}
Sep 17 17:47:18.828922 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:18.828882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2" event={"ID":"a55aa06e-fcc9-430a-af47-a148b7049736","Type":"ContainerStarted","Data":"d700863a1efbfdd51b1eb0a1f8ba9501bf6b1ed83483895987404d06dc2359b8"}
Sep 17 17:47:18.844631 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:18.844582    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-t7rt2" podStartSLOduration=2.59123641 podStartE2EDuration="2.844567374s" totalImagesPullingTime="253.330964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:18.843259935 +0000 UTC m=+730.030695783" watchObservedRunningTime="2026-09-17 17:47:18.844567374 +0000 UTC m=+730.032003222"
Sep 17 17:47:42.906964 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.906929    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"]
Sep 17 17:47:42.912897 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.912870    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:42.915592 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.915566    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:47:42.915726 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.915595    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:47:42.915726 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.915608    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 17:47:42.915726 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.915671    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 17:47:42.915907 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.915845    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:47:42.916418 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.916402    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 17:47:42.916509 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.916492    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:47:42.926305 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:42.926283    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"]
Sep 17 17:47:43.012088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.012058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4r72\" (UniqueName: \"kubernetes.io/projected/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-api-access-r4r72\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.012217 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.012103    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.012217 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.012126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.012217 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.012192    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.012352 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.012244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.112758 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.112733    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r72\" (UniqueName: \"kubernetes.io/projected/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-api-access-r4r72\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.112897 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.112774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.112897 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.112804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.112897 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.112836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.112897 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.112869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.113228 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.113206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.113629 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.113609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.113677 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.113609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.115339 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.115319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.121554 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.121526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4r72\" (UniqueName: \"kubernetes.io/projected/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-api-access-r4r72\") pod \"isvc-sklearn-batcher-predictor-77947f8859-r8jcn\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.223736 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.223668    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:47:43.499271 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.499103    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"]
Sep 17 17:47:43.501876 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:47:43.501844    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b20ea3b_28f8_424a_8cb1_11a5fb6387cb.slice/crio-62d4f666b9c109a60eae904e0c6db4add960e8d9b76a395c57ff0dd3ccfdaa96 WatchSource:0}: Error finding container 62d4f666b9c109a60eae904e0c6db4add960e8d9b76a395c57ff0dd3ccfdaa96: Status 404 returned error can't find the container with id 62d4f666b9c109a60eae904e0c6db4add960e8d9b76a395c57ff0dd3ccfdaa96
Sep 17 17:47:43.919410 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:43.919338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerStarted","Data":"62d4f666b9c109a60eae904e0c6db4add960e8d9b76a395c57ff0dd3ccfdaa96"}
Sep 17 17:47:46.931432 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:46.931254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerStarted","Data":"b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e"}
Sep 17 17:47:50.945445 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:50.945414    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerID="b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e" exitCode=0
Sep 17 17:47:50.945866 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:47:50.945464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerDied","Data":"b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e"}
Sep 17 17:48:05.006984 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:05.006951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerStarted","Data":"200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa"}
Sep 17 17:48:08.028610 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:08.028574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerStarted","Data":"39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f"}
Sep 17 17:48:11.043178 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:11.043128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerStarted","Data":"8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1"}
Sep 17 17:48:11.043613 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:11.043469    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:48:11.043613 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:11.043538    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:48:11.043613 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:11.043573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:48:11.044698 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:11.044667    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:48:11.045281 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:11.045258    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:12.046810 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:12.046772    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:48:12.047290 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:12.047265    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:12.051756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:12.051736    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:48:12.074716 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:12.074671    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podStartSLOduration=4.042291316 podStartE2EDuration="30.074654658s" totalImagesPullingTime="22.211651525s" totalInitContainerRuntime="3.820711817s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:42 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:11.076464962 +0000 UTC m=+782.263900834" watchObservedRunningTime="2026-09-17 17:48:12.074654658 +0000 UTC m=+783.262090508"
Sep 17 17:48:13.050808 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:13.050763    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:48:13.051282 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:13.051020    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:23.050763 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:23.050717    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:48:23.051259 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:23.051091    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:33.051091 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:33.050986    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:48:33.051555 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:33.051420    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:43.050992 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:43.050945    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:48:43.051466 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:43.051366    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:48:53.051389 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:53.051339    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:48:53.051888 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:48:53.051862    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:03.051700 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:03.051642    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:49:03.052243 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:03.052219    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:13.051573 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:13.051527    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:49:13.052114 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:13.052023    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:23.051666 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:23.051636    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:49:23.052065 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:23.051728    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:49:37.748679 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.748642    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"]
Sep 17 17:49:37.749134 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.748972    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" containerID="cri-o://200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa" gracePeriod=30
Sep 17 17:49:37.749134 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.749046    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" containerID="cri-o://8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1" gracePeriod=30
Sep 17 17:49:37.749134 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.749077    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy" containerID="cri-o://39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f" gracePeriod=30
Sep 17 17:49:37.917588 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.917555    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"]
Sep 17 17:49:37.920923 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.920899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:37.923833 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.923796    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 17:49:37.923974 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.923833    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 17:49:37.937874 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:37.937849    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"]
Sep 17 17:49:38.047828 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.047753    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634d5991-e020-4498-97d9-53a6a936a452-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.047828 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.047792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.047828 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.047814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.048072 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.047937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlpxr\" (UniqueName: \"kubernetes.io/projected/634d5991-e020-4498-97d9-53a6a936a452-kube-api-access-jlpxr\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.048072 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.047980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.148963 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.148935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634d5991-e020-4498-97d9-53a6a936a452-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.149113 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.148974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.149113 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.148992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.149113 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.149016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlpxr\" (UniqueName: \"kubernetes.io/projected/634d5991-e020-4498-97d9-53a6a936a452-kube-api-access-jlpxr\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.149113 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.149058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.149377 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:49:38.149168    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 17 17:49:38.149377 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:49:38.149244    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls podName:634d5991-e020-4498-97d9-53a6a936a452 nodeName:}" failed. No retries permitted until 2026-09-17 17:49:38.64922419 +0000 UTC m=+869.836660027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" (UID: "634d5991-e020-4498-97d9-53a6a936a452") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 17 17:49:38.149475 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.149396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634d5991-e020-4498-97d9-53a6a936a452-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.149825 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.149802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.149860 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.149802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.159177 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.159159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlpxr\" (UniqueName: \"kubernetes.io/projected/634d5991-e020-4498-97d9-53a6a936a452-kube-api-access-jlpxr\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.359019 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.358936    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerID="39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f" exitCode=2
Sep 17 17:49:38.359019 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.359008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerDied","Data":"39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f"}
Sep 17 17:49:38.653143 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.653050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.655321 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.655302    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:38.830609 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:38.830577    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:39.107765 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:39.107742    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"]
Sep 17 17:49:39.110505 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:49:39.110480    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634d5991_e020_4498_97d9_53a6a936a452.slice/crio-f9da243a588ecc4fd0b5009fad107d0f6eacf43ea4e2001ae0fcfa0b9aa9a633 WatchSource:0}: Error finding container f9da243a588ecc4fd0b5009fad107d0f6eacf43ea4e2001ae0fcfa0b9aa9a633: Status 404 returned error can't find the container with id f9da243a588ecc4fd0b5009fad107d0f6eacf43ea4e2001ae0fcfa0b9aa9a633
Sep 17 17:49:39.365032 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:39.364945    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerStarted","Data":"3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f"}
Sep 17 17:49:39.365032 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:39.365022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerStarted","Data":"f9da243a588ecc4fd0b5009fad107d0f6eacf43ea4e2001ae0fcfa0b9aa9a633"}
Sep 17 17:49:42.048086 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:42.048044    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 17:49:42.378056 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:42.377972    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerID="200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa" exitCode=0
Sep 17 17:49:42.378056 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:42.378044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerDied","Data":"200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa"}
Sep 17 17:49:43.051047 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:43.051001    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:49:43.051431 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:43.051265    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:44.386284 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:44.386250    2562 generic.go:356] "Generic (PLEG): container finished" podID="634d5991-e020-4498-97d9-53a6a936a452" containerID="3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f" exitCode=0
Sep 17 17:49:44.386284 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:44.386288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerDied","Data":"3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f"}
Sep 17 17:49:45.397496 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.397461    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerStarted","Data":"29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b"}
Sep 17 17:49:45.397904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.397547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerStarted","Data":"32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f"}
Sep 17 17:49:45.397904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.397571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerStarted","Data":"ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3"}
Sep 17 17:49:45.397904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.397856    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:45.398067 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.397934    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:45.398067 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.397972    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:45.399401 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.399377    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:49:45.400117 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.400092    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:45.421091 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:45.421049    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podStartSLOduration=4.132330599 podStartE2EDuration="8.42103643s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.288705831s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:45.420087844 +0000 UTC m=+876.607523693" watchObservedRunningTime="2026-09-17 17:49:45.42103643 +0000 UTC m=+876.608472280"
Sep 17 17:49:46.401891 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:46.401850    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:49:46.402367 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:46.402345    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:47.047663 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:47.047622    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 17:49:51.407320 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:51.407294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:49:51.407893 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:51.407861    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:49:51.408081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:51.408060    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:52.048055 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:52.048008    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 17:49:52.048233 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:52.048149    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:49:53.050782 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:53.050729    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:49:53.051219 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:53.051068    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:49:57.048059 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:49:57.048015    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 17:50:01.408336 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:01.408255    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:50:01.408783 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:01.408753    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:02.047406 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:02.047362    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 17:50:03.051739 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:03.051688    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 17:50:03.052203 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:03.051878    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:50:03.052203 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:03.051935    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:03.052203 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:03.052183    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:50:07.047311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.047263    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 17:50:07.893656 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.893631    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:50:07.961580 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.961549    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-cabundle-cert\") pod \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") "
Sep 17 17:50:07.961745 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.961617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kserve-provision-location\") pod \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") "
Sep 17 17:50:07.961745 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.961639    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-proxy-tls\") pod \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") "
Sep 17 17:50:07.961745 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.961663    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4r72\" (UniqueName: \"kubernetes.io/projected/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-api-access-r4r72\") pod \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") "
Sep 17 17:50:07.961745 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.961686    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-rbac-proxy-sar-config\") pod \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\" (UID: \"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb\") "
Sep 17 17:50:07.962026 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.961996    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" (UID: "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:07.962026 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.962020    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" (UID: "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:07.962175 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.962088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" (UID: "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:07.963827 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.963806    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-api-access-r4r72" (OuterVolumeSpecName: "kube-api-access-r4r72") pod "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" (UID: "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb"). InnerVolumeSpecName "kube-api-access-r4r72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:07.963931 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:07.963896    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" (UID: "9b20ea3b-28f8-424a-8cb1-11a5fb6387cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:08.063156 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.063062    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.063156 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.063103    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.063156 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.063114    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4r72\" (UniqueName: \"kubernetes.io/projected/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-api-access-r4r72\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.063156 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.063122    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.063156 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.063139    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:50:08.487072 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.487046    2562 generic.go:356] "Generic (PLEG): container finished" podID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerID="8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1" exitCode=0
Sep 17 17:50:08.487218 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.487102    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"
Sep 17 17:50:08.487218 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.487133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerDied","Data":"8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1"}
Sep 17 17:50:08.487218 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.487198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn" event={"ID":"9b20ea3b-28f8-424a-8cb1-11a5fb6387cb","Type":"ContainerDied","Data":"62d4f666b9c109a60eae904e0c6db4add960e8d9b76a395c57ff0dd3ccfdaa96"}
Sep 17 17:50:08.487314 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.487229    2562 scope.go:117] "RemoveContainer" containerID="8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1"
Sep 17 17:50:08.495676 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.495657    2562 scope.go:117] "RemoveContainer" containerID="39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f"
Sep 17 17:50:08.502865 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.502846    2562 scope.go:117] "RemoveContainer" containerID="200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa"
Sep 17 17:50:08.510070 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.510022    2562 scope.go:117] "RemoveContainer" containerID="b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e"
Sep 17 17:50:08.510794 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.510774    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"]
Sep 17 17:50:08.515283 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.515262    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-77947f8859-r8jcn"]
Sep 17 17:50:08.517643 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.517627    2562 scope.go:117] "RemoveContainer" containerID="8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1"
Sep 17 17:50:08.517879 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:50:08.517861    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1\": container with ID starting with 8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1 not found: ID does not exist" containerID="8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1"
Sep 17 17:50:08.517942 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.517889    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1"} err="failed to get container status \"8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1\": rpc error: code = NotFound desc = could not find container \"8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1\": container with ID starting with 8a8e7361fafee46baf210b5028d0f765100e0d6172594fd5a348f5a56971d9e1 not found: ID does not exist"
Sep 17 17:50:08.517942 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.517907    2562 scope.go:117] "RemoveContainer" containerID="39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f"
Sep 17 17:50:08.518174 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:50:08.518157    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f\": container with ID starting with 39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f not found: ID does not exist" containerID="39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f"
Sep 17 17:50:08.518216 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.518178    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f"} err="failed to get container status \"39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f\": rpc error: code = NotFound desc = could not find container \"39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f\": container with ID starting with 39fde7c7738bcf7690ba66e7804c1dda0bbdcab1d11c798178218ba4ee45286f not found: ID does not exist"
Sep 17 17:50:08.518216 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.518192    2562 scope.go:117] "RemoveContainer" containerID="200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa"
Sep 17 17:50:08.518386 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:50:08.518373    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa\": container with ID starting with 200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa not found: ID does not exist" containerID="200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa"
Sep 17 17:50:08.518429 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.518389    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa"} err="failed to get container status \"200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa\": rpc error: code = NotFound desc = could not find container \"200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa\": container with ID starting with 200c33d69c7191c67b990c67902ce04a59b58b4fbe0534fea45c6805ed16b8aa not found: ID does not exist"
Sep 17 17:50:08.518429 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.518400    2562 scope.go:117] "RemoveContainer" containerID="b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e"
Sep 17 17:50:08.518651 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:50:08.518635    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e\": container with ID starting with b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e not found: ID does not exist" containerID="b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e"
Sep 17 17:50:08.518689 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:08.518655    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e"} err="failed to get container status \"b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e\": rpc error: code = NotFound desc = could not find container \"b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e\": container with ID starting with b2d892981cb9f849f92a91196eb56117f3b0a89b2f49f1e08e66d8efdcecee1e not found: ID does not exist"
Sep 17 17:50:09.276648 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:09.276618    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" path="/var/lib/kubelet/pods/9b20ea3b-28f8-424a-8cb1-11a5fb6387cb/volumes"
Sep 17 17:50:11.408605 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:11.408557    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:50:11.409005 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:11.408904    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:21.407757 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:21.407716    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:50:21.408237 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:21.408072    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:31.408164 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:31.408121    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:50:31.408615 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:31.408594    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:41.408217 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:41.408170    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:50:41.408694 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:41.408674    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:50:51.408244 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:51.408199    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:50:51.410685 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:50:51.408634    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:01.409126 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:01.409095    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:51:01.409585 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:01.409187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:51:12.933343 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:12.933297    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"]
Sep 17 17:51:12.933808 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:12.933636    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" containerID="cri-o://29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b" gracePeriod=30
Sep 17 17:51:12.933808 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:12.933662    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy" containerID="cri-o://32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f" gracePeriod=30
Sep 17 17:51:12.933808 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:12.933624    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" containerID="cri-o://ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3" gracePeriod=30
Sep 17 17:51:13.036398 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036366    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"]
Sep 17 17:51:13.036626 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036614    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container"
Sep 17 17:51:13.036678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036627    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container"
Sep 17 17:51:13.036678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036636    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy"
Sep 17 17:51:13.036678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036642    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy"
Sep 17 17:51:13.036678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036651    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent"
Sep 17 17:51:13.036678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036657    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent"
Sep 17 17:51:13.036678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036662    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="storage-initializer"
Sep 17 17:51:13.036678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036668    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="storage-initializer"
Sep 17 17:51:13.036884 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036718    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kube-rbac-proxy"
Sep 17 17:51:13.036884 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036725    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="agent"
Sep 17 17:51:13.036884 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.036731    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b20ea3b-28f8-424a-8cb1-11a5fb6387cb" containerName="kserve-container"
Sep 17 17:51:13.039749 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.039718    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.042488 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.042470    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 17:51:13.042881 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.042867    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 17:51:13.052682 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.052661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"]
Sep 17 17:51:13.129184 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.129149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sfr6\" (UniqueName: \"kubernetes.io/projected/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-api-access-5sfr6\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.129365 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.129197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.129365 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.129217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.230592 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.230501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sfr6\" (UniqueName: \"kubernetes.io/projected/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-api-access-5sfr6\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.230592 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.230553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.230816 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.230590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.230816 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:13.230718    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 17:51:13.230816 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:13.230787    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls podName:ed3d0bac-bc26-4e1c-bb7d-f0abd611392e nodeName:}" failed. No retries permitted until 2026-09-17 17:51:13.730765836 +0000 UTC m=+964.918201665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-qhsvk" (UID: "ed3d0bac-bc26-4e1c-bb7d-f0abd611392e") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 17:51:13.231336 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.231315    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.239171 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.239150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sfr6\" (UniqueName: \"kubernetes.io/projected/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-api-access-5sfr6\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.718307 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.718273    2562 generic.go:356] "Generic (PLEG): container finished" podID="634d5991-e020-4498-97d9-53a6a936a452" containerID="32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f" exitCode=2
Sep 17 17:51:13.718483 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.718357    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerDied","Data":"32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f"}
Sep 17 17:51:13.734633 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.734605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.737036 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.737019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-qhsvk\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:13.949597 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:13.949562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:14.220206 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:14.220173    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"]
Sep 17 17:51:14.223289 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:51:14.223260    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5 WatchSource:0}: Error finding container 0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5: Status 404 returned error can't find the container with id 0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5
Sep 17 17:51:14.225245 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:14.225228    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:51:14.723975 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:14.723943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" event={"ID":"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e","Type":"ContainerStarted","Data":"0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5"}
Sep 17 17:51:15.731835 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:15.731757    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" event={"ID":"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e","Type":"ContainerStarted","Data":"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"}
Sep 17 17:51:15.731835 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:15.731810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" event={"ID":"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e","Type":"ContainerStarted","Data":"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"}
Sep 17 17:51:15.732272 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:15.732031    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:15.756074 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:15.756029    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" podStartSLOduration=1.612152208 podStartE2EDuration="2.756013505s" totalImagesPullingTime="1.143861297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:15.753419401 +0000 UTC m=+966.940855260" watchObservedRunningTime="2026-09-17 17:51:15.756013505 +0000 UTC m=+966.943449354"
Sep 17 17:51:16.402754 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:16.402712    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:51:16.735834 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:16.735739    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:16.737668 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:16.737649    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:17.741383 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:17.741349    2562 generic.go:356] "Generic (PLEG): container finished" podID="634d5991-e020-4498-97d9-53a6a936a452" containerID="ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3" exitCode=0
Sep 17 17:51:17.741740 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:17.741418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerDied","Data":"ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3"}
Sep 17 17:51:21.402624 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:21.402581    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:51:21.408022 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:21.407986    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:51:21.408281 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:21.408263    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:23.749767 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:23.749739    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:51:26.402879 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:26.402833    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:51:26.403288 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:26.402987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:51:31.402971 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:31.402867    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:51:31.408256 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:31.408232    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:51:31.408628 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:31.408601    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:33.150088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.150054    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"]
Sep 17 17:51:33.153263 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.153247    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.155836 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.155815    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 17:51:33.155836 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.155831    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 17:51:33.166215 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.166196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-cabundle-cert\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.166328 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.166229    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd00201-829d-4766-b66d-321c3e4bc637-kserve-provision-location\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.166328 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.166247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbhr\" (UniqueName: \"kubernetes.io/projected/2bd00201-829d-4766-b66d-321c3e4bc637-kube-api-access-kpbhr\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.166442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.166338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.166442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.166396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.170901 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.170882    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"]
Sep 17 17:51:33.267266 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.267230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.267425 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.267278    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.267425 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.267314    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-cabundle-cert\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.267425 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.267342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd00201-829d-4766-b66d-321c3e4bc637-kserve-provision-location\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.267425 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.267366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbhr\" (UniqueName: \"kubernetes.io/projected/2bd00201-829d-4766-b66d-321c3e4bc637-kube-api-access-kpbhr\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.267425 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:33.267370    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 17 17:51:33.267652 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:33.267480    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls podName:2bd00201-829d-4766-b66d-321c3e4bc637 nodeName:}" failed. No retries permitted until 2026-09-17 17:51:33.767461296 +0000 UTC m=+984.954897123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls") pod "isvc-logger-predictor-7d8d566b5b-tm7sf" (UID: "2bd00201-829d-4766-b66d-321c3e4bc637") : secret "isvc-logger-predictor-serving-cert" not found
Sep 17 17:51:33.267740 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.267716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd00201-829d-4766-b66d-321c3e4bc637-kserve-provision-location\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.268033 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.268014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-cabundle-cert\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.268183 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.268116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.277341 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.277316    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbhr\" (UniqueName: \"kubernetes.io/projected/2bd00201-829d-4766-b66d-321c3e4bc637-kube-api-access-kpbhr\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.769817 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.769785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:33.772180 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:33.772160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls\") pod \"isvc-logger-predictor-7d8d566b5b-tm7sf\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:34.063399 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:34.063324    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:34.341459 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:34.341380    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"]
Sep 17 17:51:34.344264 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:51:34.344240    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd00201_829d_4766_b66d_321c3e4bc637.slice/crio-d8801ab0853d78cfb53bb631265ef3896e67d0a8ac9cbf1ace01ed2a2647aa57 WatchSource:0}: Error finding container d8801ab0853d78cfb53bb631265ef3896e67d0a8ac9cbf1ace01ed2a2647aa57: Status 404 returned error can't find the container with id d8801ab0853d78cfb53bb631265ef3896e67d0a8ac9cbf1ace01ed2a2647aa57
Sep 17 17:51:34.801016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:34.800986    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerStarted","Data":"61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c"}
Sep 17 17:51:34.801167 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:34.801047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerStarted","Data":"d8801ab0853d78cfb53bb631265ef3896e67d0a8ac9cbf1ace01ed2a2647aa57"}
Sep 17 17:51:36.402669 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:36.402633    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:51:38.818115 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:38.818082    2562 generic.go:356] "Generic (PLEG): container finished" podID="2bd00201-829d-4766-b66d-321c3e4bc637" containerID="61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c" exitCode=0
Sep 17 17:51:38.818551 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:38.818125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerDied","Data":"61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c"}
Sep 17 17:51:39.824459 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:39.824424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerStarted","Data":"464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a"}
Sep 17 17:51:39.824818 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:39.824492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerStarted","Data":"72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4"}
Sep 17 17:51:39.824818 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:39.824518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerStarted","Data":"0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e"}
Sep 17 17:51:39.824818 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:39.824792    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:39.825059 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:39.824945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:39.826229 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:39.826204    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:51:39.848959 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:39.848901    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podStartSLOduration=3.081753604 podStartE2EDuration="6.848887612s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767134008s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:39.847558047 +0000 UTC m=+991.034993895" watchObservedRunningTime="2026-09-17 17:51:39.848887612 +0000 UTC m=+991.036323460"
Sep 17 17:51:40.828086 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:40.828055    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:40.828442 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:40.828173    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:51:40.829224 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:40.829197    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:41.402099 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:41.402057    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 17:51:41.408480 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:41.408454    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:5000: connect: connection refused"
Sep 17 17:51:41.408618 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:41.408561    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:51:41.408822 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:41.408804    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:41.408937 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:41.408896    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:51:41.831965 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:41.831905    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:51:41.832337 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:41.832282    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:43.066761 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.066740    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:51:43.135381 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135352    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls\") pod \"634d5991-e020-4498-97d9-53a6a936a452\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") "
Sep 17 17:51:43.135546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135396    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-kube-rbac-proxy-sar-config\") pod \"634d5991-e020-4498-97d9-53a6a936a452\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") "
Sep 17 17:51:43.135546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlpxr\" (UniqueName: \"kubernetes.io/projected/634d5991-e020-4498-97d9-53a6a936a452-kube-api-access-jlpxr\") pod \"634d5991-e020-4498-97d9-53a6a936a452\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") "
Sep 17 17:51:43.135546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135442    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-cabundle-cert\") pod \"634d5991-e020-4498-97d9-53a6a936a452\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") "
Sep 17 17:51:43.135546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135520    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634d5991-e020-4498-97d9-53a6a936a452-kserve-provision-location\") pod \"634d5991-e020-4498-97d9-53a6a936a452\" (UID: \"634d5991-e020-4498-97d9-53a6a936a452\") "
Sep 17 17:51:43.135805 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135773    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "634d5991-e020-4498-97d9-53a6a936a452" (UID: "634d5991-e020-4498-97d9-53a6a936a452"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:43.135889 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135817    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "634d5991-e020-4498-97d9-53a6a936a452" (UID: "634d5991-e020-4498-97d9-53a6a936a452"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:43.135889 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.135870    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634d5991-e020-4498-97d9-53a6a936a452-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "634d5991-e020-4498-97d9-53a6a936a452" (UID: "634d5991-e020-4498-97d9-53a6a936a452"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:51:43.138035 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.138014    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634d5991-e020-4498-97d9-53a6a936a452-kube-api-access-jlpxr" (OuterVolumeSpecName: "kube-api-access-jlpxr") pod "634d5991-e020-4498-97d9-53a6a936a452" (UID: "634d5991-e020-4498-97d9-53a6a936a452"). InnerVolumeSpecName "kube-api-access-jlpxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:43.138035 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.138024    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "634d5991-e020-4498-97d9-53a6a936a452" (UID: "634d5991-e020-4498-97d9-53a6a936a452"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:43.236500 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.236473    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/634d5991-e020-4498-97d9-53a6a936a452-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.236500 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.236499    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/634d5991-e020-4498-97d9-53a6a936a452-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.236681 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.236509    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.236681 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.236522    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlpxr\" (UniqueName: \"kubernetes.io/projected/634d5991-e020-4498-97d9-53a6a936a452-kube-api-access-jlpxr\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.236681 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.236531    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/634d5991-e020-4498-97d9-53a6a936a452-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:51:43.841989 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.841954    2562 generic.go:356] "Generic (PLEG): container finished" podID="634d5991-e020-4498-97d9-53a6a936a452" containerID="29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b" exitCode=0
Sep 17 17:51:43.842157 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.842012    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"
Sep 17 17:51:43.842157 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.842034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerDied","Data":"29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b"}
Sep 17 17:51:43.842157 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.842098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb" event={"ID":"634d5991-e020-4498-97d9-53a6a936a452","Type":"ContainerDied","Data":"f9da243a588ecc4fd0b5009fad107d0f6eacf43ea4e2001ae0fcfa0b9aa9a633"}
Sep 17 17:51:43.842157 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.842129    2562 scope.go:117] "RemoveContainer" containerID="29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b"
Sep 17 17:51:43.850116 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.850098    2562 scope.go:117] "RemoveContainer" containerID="32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f"
Sep 17 17:51:43.862725 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.862707    2562 scope.go:117] "RemoveContainer" containerID="ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3"
Sep 17 17:51:43.867112 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.867091    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"]
Sep 17 17:51:43.870151 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.870127    2562 scope.go:117] "RemoveContainer" containerID="3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f"
Sep 17 17:51:43.871185 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.871168    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-77cb8588d9-9l9tb"]
Sep 17 17:51:43.877461 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.877447    2562 scope.go:117] "RemoveContainer" containerID="29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b"
Sep 17 17:51:43.877686 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:43.877663    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b\": container with ID starting with 29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b not found: ID does not exist" containerID="29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b"
Sep 17 17:51:43.877739 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.877694    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b"} err="failed to get container status \"29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b\": rpc error: code = NotFound desc = could not find container \"29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b\": container with ID starting with 29dc58169d81842f116bc4e5f792bce3a75216b2e1f3a371b208e47250f53f9b not found: ID does not exist"
Sep 17 17:51:43.877739 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.877713    2562 scope.go:117] "RemoveContainer" containerID="32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f"
Sep 17 17:51:43.877986 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:43.877968    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f\": container with ID starting with 32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f not found: ID does not exist" containerID="32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f"
Sep 17 17:51:43.878039 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.877993    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f"} err="failed to get container status \"32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f\": rpc error: code = NotFound desc = could not find container \"32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f\": container with ID starting with 32fdf2bd0049e6568fbe5a72f509583e8c300225e990f220885c4956dfeaea1f not found: ID does not exist"
Sep 17 17:51:43.878039 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.878014    2562 scope.go:117] "RemoveContainer" containerID="ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3"
Sep 17 17:51:43.878227 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:43.878210    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3\": container with ID starting with ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3 not found: ID does not exist" containerID="ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3"
Sep 17 17:51:43.878272 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.878232    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3"} err="failed to get container status \"ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3\": rpc error: code = NotFound desc = could not find container \"ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3\": container with ID starting with ce6d2ad4952eebbe6d573a459222428347d92afb6cd27771e061bd5ab51889a3 not found: ID does not exist"
Sep 17 17:51:43.878272 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.878250    2562 scope.go:117] "RemoveContainer" containerID="3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f"
Sep 17 17:51:43.878500 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:51:43.878484    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f\": container with ID starting with 3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f not found: ID does not exist" containerID="3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f"
Sep 17 17:51:43.878541 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:43.878505    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f"} err="failed to get container status \"3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f\": rpc error: code = NotFound desc = could not find container \"3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f\": container with ID starting with 3d5d0dbad8818602d3d03a0204adb12fb497b830654fc6da00cfb09b8ea5449f not found: ID does not exist"
Sep 17 17:51:45.275638 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:45.275603    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="634d5991-e020-4498-97d9-53a6a936a452" path="/var/lib/kubelet/pods/634d5991-e020-4498-97d9-53a6a936a452/volumes"
Sep 17 17:51:46.835959 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:46.835928    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:51:46.836471 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:46.836443    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:51:46.836779 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:46.836755    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:51:56.837119 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:56.837073    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:51:56.837597 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:51:56.837569    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:06.837149 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:06.837109    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:52:06.837606 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:06.837577    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:16.836462 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:16.836422    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:52:16.836886 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:16.836863    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:26.836931 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:26.836875    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:52:26.837366 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:26.837344    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:36.836391 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:36.836349    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:52:36.836863 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:36.836831    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:46.837081 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:46.837035    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:52:46.837589 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:46.837541    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:52:53.276209 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:53.276182    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:52:53.276649 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:52:53.276255    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:53:08.096826 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.096791    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-qhsvk_ed3d0bac-bc26-4e1c-bb7d-f0abd611392e/kserve-container/0.log"
Sep 17 17:53:08.470338 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.470262    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"]
Sep 17 17:53:08.470536 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.470515    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" containerID="cri-o://0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e" gracePeriod=30
Sep 17 17:53:08.470610 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.470545    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" containerID="cri-o://464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a" gracePeriod=30
Sep 17 17:53:08.470664 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.470585    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" containerID="cri-o://72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4" gracePeriod=30
Sep 17 17:53:08.478005 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.477984    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"]
Sep 17 17:53:08.478194 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.478178    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kserve-container" containerID="cri-o://63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce" gracePeriod=30
Sep 17 17:53:08.478282 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.478261    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kube-rbac-proxy" containerID="cri-o://46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9" gracePeriod=30
Sep 17 17:53:08.710937 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.710899    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:53:08.827702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.827671    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-rbac-proxy-sar-config\") pod \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") "
Sep 17 17:53:08.827860 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.827718    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls\") pod \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") "
Sep 17 17:53:08.827860 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.827741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sfr6\" (UniqueName: \"kubernetes.io/projected/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-api-access-5sfr6\") pod \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\" (UID: \"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e\") "
Sep 17 17:53:08.828159 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.828127    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" (UID: "ed3d0bac-bc26-4e1c-bb7d-f0abd611392e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:08.829900 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.829873    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" (UID: "ed3d0bac-bc26-4e1c-bb7d-f0abd611392e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:08.829974 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.829942    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-api-access-5sfr6" (OuterVolumeSpecName: "kube-api-access-5sfr6") pod "ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" (UID: "ed3d0bac-bc26-4e1c-bb7d-f0abd611392e"). InnerVolumeSpecName "kube-api-access-5sfr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:08.929155 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.929108    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:08.929155 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.929150    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:08.929155 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:08.929160    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sfr6\" (UniqueName: \"kubernetes.io/projected/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e-kube-api-access-5sfr6\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:09.149212 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.149126    2562 generic.go:356] "Generic (PLEG): container finished" podID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerID="46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9" exitCode=2
Sep 17 17:53:09.149212 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.149153    2562 generic.go:356] "Generic (PLEG): container finished" podID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerID="63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce" exitCode=2
Sep 17 17:53:09.149212 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.149181    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"
Sep 17 17:53:09.149757 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.149221    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" event={"ID":"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e","Type":"ContainerDied","Data":"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"}
Sep 17 17:53:09.149757 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.149298    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" event={"ID":"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e","Type":"ContainerDied","Data":"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"}
Sep 17 17:53:09.149757 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.149330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk" event={"ID":"ed3d0bac-bc26-4e1c-bb7d-f0abd611392e","Type":"ContainerDied","Data":"0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5"}
Sep 17 17:53:09.149757 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.149366    2562 scope.go:117] "RemoveContainer" containerID="46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"
Sep 17 17:53:09.152116 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.152091    2562 generic.go:356] "Generic (PLEG): container finished" podID="2bd00201-829d-4766-b66d-321c3e4bc637" containerID="72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4" exitCode=2
Sep 17 17:53:09.152233 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.152139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerDied","Data":"72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4"}
Sep 17 17:53:09.157729 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.157714    2562 scope.go:117] "RemoveContainer" containerID="63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"
Sep 17 17:53:09.165037 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.165001    2562 scope.go:117] "RemoveContainer" containerID="46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"
Sep 17 17:53:09.165297 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:09.165281    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9\": container with ID starting with 46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9 not found: ID does not exist" containerID="46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"
Sep 17 17:53:09.165339 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.165306    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"} err="failed to get container status \"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9\": rpc error: code = NotFound desc = could not find container \"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9\": container with ID starting with 46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9 not found: ID does not exist"
Sep 17 17:53:09.165339 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.165329    2562 scope.go:117] "RemoveContainer" containerID="63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"
Sep 17 17:53:09.165583 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:09.165567    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce\": container with ID starting with 63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce not found: ID does not exist" containerID="63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"
Sep 17 17:53:09.165623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.165590    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"} err="failed to get container status \"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce\": rpc error: code = NotFound desc = could not find container \"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce\": container with ID starting with 63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce not found: ID does not exist"
Sep 17 17:53:09.165623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.165606    2562 scope.go:117] "RemoveContainer" containerID="46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"
Sep 17 17:53:09.165868 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.165847    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9"} err="failed to get container status \"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9\": rpc error: code = NotFound desc = could not find container \"46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9\": container with ID starting with 46dc977db328408bf10d03e182b0376d2ed69dc512dc5822d801f27cfa0a44b9 not found: ID does not exist"
Sep 17 17:53:09.165868 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.165867    2562 scope.go:117] "RemoveContainer" containerID="63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"
Sep 17 17:53:09.166132 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.166111    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce"} err="failed to get container status \"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce\": rpc error: code = NotFound desc = could not find container \"63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce\": container with ID starting with 63e5a6a9c390eeae50285cffdc4433180d3f426a9826f04affbef27d444fbdce not found: ID does not exist"
Sep 17 17:53:09.179820 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.179793    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"]
Sep 17 17:53:09.183723 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.183703    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-qhsvk"]
Sep 17 17:53:09.275477 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:09.275445    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" path="/var/lib/kubelet/pods/ed3d0bac-bc26-4e1c-bb7d-f0abd611392e/volumes"
Sep 17 17:53:10.122326 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:10.122289    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:10.122532 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:10.122323    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:11.049743 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:11.049707    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:11.832943 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:11.832879    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:13.167177 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:13.167096    2562 generic.go:356] "Generic (PLEG): container finished" podID="2bd00201-829d-4766-b66d-321c3e4bc637" containerID="0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e" exitCode=0
Sep 17 17:53:13.167177 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:13.167139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerDied","Data":"0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e"}
Sep 17 17:53:13.272152 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:13.272113    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:53:13.272477 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:13.272456    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:16.832762 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:16.832721    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:19.066138 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066103    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"]
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066407    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="storage-initializer"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066421    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="storage-initializer"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066433    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066439    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066445    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066450    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066458    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kserve-container"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066463    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kserve-container"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066471    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066476    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066482    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kube-rbac-proxy"
Sep 17 17:53:19.066486 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066488    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kube-rbac-proxy"
Sep 17 17:53:19.066862 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066532    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="agent"
Sep 17 17:53:19.066862 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066539    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kserve-container"
Sep 17 17:53:19.066862 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066545    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="634d5991-e020-4498-97d9-53a6a936a452" containerName="kube-rbac-proxy"
Sep 17 17:53:19.066862 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066551    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kube-rbac-proxy"
Sep 17 17:53:19.066862 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.066557    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3d0bac-bc26-4e1c-bb7d-f0abd611392e" containerName="kserve-container"
Sep 17 17:53:19.069500 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.069485    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.072179 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.072155    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:53:19.072283 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.072194    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 17:53:19.081818 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.081798    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"]
Sep 17 17:53:19.202446 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.202412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491722a-62fc-4174-bbd7-99ca26917063-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.202599 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.202467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.202599 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.202540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.202599 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.202570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7z4\" (UniqueName: \"kubernetes.io/projected/c491722a-62fc-4174-bbd7-99ca26917063-kube-api-access-gz7z4\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.202599 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.202595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.303669 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.303639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491722a-62fc-4174-bbd7-99ca26917063-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.303813 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.303686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.303813 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.303714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.303813 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.303746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7z4\" (UniqueName: \"kubernetes.io/projected/c491722a-62fc-4174-bbd7-99ca26917063-kube-api-access-gz7z4\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.303813 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.303775    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.304019 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:19.303891    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:53:19.304019 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:19.303989    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls podName:c491722a-62fc-4174-bbd7-99ca26917063 nodeName:}" failed. No retries permitted until 2026-09-17 17:53:19.803967331 +0000 UTC m=+1090.991403160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" (UID: "c491722a-62fc-4174-bbd7-99ca26917063") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 17:53:19.304136 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.304119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491722a-62fc-4174-bbd7-99ca26917063-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.304422 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.304404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.304459 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.304424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.314690 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.314665    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7z4\" (UniqueName: \"kubernetes.io/projected/c491722a-62fc-4174-bbd7-99ca26917063-kube-api-access-gz7z4\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.807262 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.807224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.809591 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.809571    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:19.874362 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:19.874329    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:19.979326 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:19.979296    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:20.257567 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:20.257538    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"]
Sep 17 17:53:20.259774 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:53:20.259742    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc491722a_62fc_4174_bbd7_99ca26917063.slice/crio-ead3157001f923705cb8de2850e571f78d2738e4e12a3905066bfb43ee967e86 WatchSource:0}: Error finding container ead3157001f923705cb8de2850e571f78d2738e4e12a3905066bfb43ee967e86: Status 404 returned error can't find the container with id ead3157001f923705cb8de2850e571f78d2738e4e12a3905066bfb43ee967e86
Sep 17 17:53:21.057617 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:21.057583    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:21.201980 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:21.201944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerStarted","Data":"c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145"}
Sep 17 17:53:21.202168 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:21.202024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerStarted","Data":"ead3157001f923705cb8de2850e571f78d2738e4e12a3905066bfb43ee967e86"}
Sep 17 17:53:21.832421 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:21.832381    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:21.832842 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:21.832496    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:53:23.272139 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:23.272095    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:53:23.272577 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:23.272462    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:25.218823 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:25.218788    2562 generic.go:356] "Generic (PLEG): container finished" podID="c491722a-62fc-4174-bbd7-99ca26917063" containerID="c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145" exitCode=0
Sep 17 17:53:25.219312 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:25.218863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerDied","Data":"c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145"}
Sep 17 17:53:26.832707 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:26.832667    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:31.100671 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:31.100625    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:31.832714 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:31.832670    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:33.272753 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:33.272703    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:53:33.273243 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:33.273215    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 17:53:33.276606 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:33.276586    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:53:33.276735 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:33.276666    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:53:34.850752 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:34.850684    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:36.832743 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:36.832705    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:41.138202 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:41.138165    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:41.832600 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:41.832555    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:43.272999 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:43.272948    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Sep 17 17:53:43.272999 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:43.272976    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent" probeResult="failure" output="Get \"http://10.132.0.22:9081/\": dial tcp 10.132.0.22:9081: connect: connection refused"
Sep 17 17:53:46.832756 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:46.832711    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Sep 17 17:53:46.988185 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:46.988161    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:53:47.031793 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.031767    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd00201-829d-4766-b66d-321c3e4bc637-kserve-provision-location\") pod \"2bd00201-829d-4766-b66d-321c3e4bc637\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") "
Sep 17 17:53:47.031961 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.031836    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-cabundle-cert\") pod \"2bd00201-829d-4766-b66d-321c3e4bc637\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") "
Sep 17 17:53:47.031961 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.031856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpbhr\" (UniqueName: \"kubernetes.io/projected/2bd00201-829d-4766-b66d-321c3e4bc637-kube-api-access-kpbhr\") pod \"2bd00201-829d-4766-b66d-321c3e4bc637\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") "
Sep 17 17:53:47.031961 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.031886    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls\") pod \"2bd00201-829d-4766-b66d-321c3e4bc637\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") "
Sep 17 17:53:47.031961 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.031939    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-kube-rbac-proxy-sar-config\") pod \"2bd00201-829d-4766-b66d-321c3e4bc637\" (UID: \"2bd00201-829d-4766-b66d-321c3e4bc637\") "
Sep 17 17:53:47.032173 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.032102    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd00201-829d-4766-b66d-321c3e4bc637-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bd00201-829d-4766-b66d-321c3e4bc637" (UID: "2bd00201-829d-4766-b66d-321c3e4bc637"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:53:47.032289 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.032233    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bd00201-829d-4766-b66d-321c3e4bc637" (UID: "2bd00201-829d-4766-b66d-321c3e4bc637"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:47.032430 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.032408    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2bd00201-829d-4766-b66d-321c3e4bc637" (UID: "2bd00201-829d-4766-b66d-321c3e4bc637"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:53:47.033886 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.033854    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd00201-829d-4766-b66d-321c3e4bc637-kube-api-access-kpbhr" (OuterVolumeSpecName: "kube-api-access-kpbhr") pod "2bd00201-829d-4766-b66d-321c3e4bc637" (UID: "2bd00201-829d-4766-b66d-321c3e4bc637"). InnerVolumeSpecName "kube-api-access-kpbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:53:47.033995 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.033943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bd00201-829d-4766-b66d-321c3e4bc637" (UID: "2bd00201-829d-4766-b66d-321c3e4bc637"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:53:47.132871 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.132833    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.132871 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.132871    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpbhr\" (UniqueName: \"kubernetes.io/projected/2bd00201-829d-4766-b66d-321c3e4bc637-kube-api-access-kpbhr\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.133122 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.132885    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bd00201-829d-4766-b66d-321c3e4bc637-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.133122 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.132902    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bd00201-829d-4766-b66d-321c3e4bc637-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.133122 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.132930    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bd00201-829d-4766-b66d-321c3e4bc637-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:53:47.318880 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.318815    2562 generic.go:356] "Generic (PLEG): container finished" podID="2bd00201-829d-4766-b66d-321c3e4bc637" containerID="464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a" exitCode=137
Sep 17 17:53:47.319016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.318879    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"
Sep 17 17:53:47.319016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.318895    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerDied","Data":"464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a"}
Sep 17 17:53:47.319016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.318976    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf" event={"ID":"2bd00201-829d-4766-b66d-321c3e4bc637","Type":"ContainerDied","Data":"d8801ab0853d78cfb53bb631265ef3896e67d0a8ac9cbf1ace01ed2a2647aa57"}
Sep 17 17:53:47.319016 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.319009    2562 scope.go:117] "RemoveContainer" containerID="464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a"
Sep 17 17:53:47.337661 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.337639    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"]
Sep 17 17:53:47.341729 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.341706    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-7d8d566b5b-tm7sf"]
Sep 17 17:53:47.463593 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.463572    2562 scope.go:117] "RemoveContainer" containerID="72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4"
Sep 17 17:53:47.564105 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.564081    2562 scope.go:117] "RemoveContainer" containerID="0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e"
Sep 17 17:53:47.571710 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.571690    2562 scope.go:117] "RemoveContainer" containerID="61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c"
Sep 17 17:53:47.578789 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.578772    2562 scope.go:117] "RemoveContainer" containerID="464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a"
Sep 17 17:53:47.579097 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:47.579074    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a\": container with ID starting with 464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a not found: ID does not exist" containerID="464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a"
Sep 17 17:53:47.579172 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.579102    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a"} err="failed to get container status \"464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a\": rpc error: code = NotFound desc = could not find container \"464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a\": container with ID starting with 464491555b26031c22655cc9be9bd5022cd855741ebc2f233d3583a172cd389a not found: ID does not exist"
Sep 17 17:53:47.579172 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.579140    2562 scope.go:117] "RemoveContainer" containerID="72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4"
Sep 17 17:53:47.579438 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:47.579423    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4\": container with ID starting with 72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4 not found: ID does not exist" containerID="72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4"
Sep 17 17:53:47.579480 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.579444    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4"} err="failed to get container status \"72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4\": rpc error: code = NotFound desc = could not find container \"72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4\": container with ID starting with 72f072b6703ff6cc15d78a3e7f3e6f41f651248178aba44233430549d6d2e8e4 not found: ID does not exist"
Sep 17 17:53:47.579480 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.579458    2562 scope.go:117] "RemoveContainer" containerID="0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e"
Sep 17 17:53:47.579694 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:47.579680    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e\": container with ID starting with 0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e not found: ID does not exist" containerID="0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e"
Sep 17 17:53:47.579774 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.579697    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e"} err="failed to get container status \"0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e\": rpc error: code = NotFound desc = could not find container \"0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e\": container with ID starting with 0e9f06c8c9fa5d072bcf4403707946a18ba027256dd50244c6042bb0e518040e not found: ID does not exist"
Sep 17 17:53:47.579774 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.579718    2562 scope.go:117] "RemoveContainer" containerID="61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c"
Sep 17 17:53:47.579945 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:47.579924    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c\": container with ID starting with 61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c not found: ID does not exist" containerID="61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c"
Sep 17 17:53:47.580046 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:47.579945    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c"} err="failed to get container status \"61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c\": rpc error: code = NotFound desc = could not find container \"61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c\": container with ID starting with 61b8bcc95c0aa6be7abe2b610847ec5a5d5c5d894b3effeb6fe2975c5eead57c not found: ID does not exist"
Sep 17 17:53:48.325702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:48.325672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerStarted","Data":"eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0"}
Sep 17 17:53:48.326088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:48.325733    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerStarted","Data":"78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d"}
Sep 17 17:53:48.326088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:48.325938    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:48.347952 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:48.347890    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podStartSLOduration=2.652934445 podStartE2EDuration="29.347875698s" totalImagesPullingTime="22.744668777s" totalInitContainerRuntime="3.950272476s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:53:48.346826298 +0000 UTC m=+1119.534262169" watchObservedRunningTime="2026-09-17 17:53:48.347875698 +0000 UTC m=+1119.535311613"
Sep 17 17:53:49.276444 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:49.276405    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" path="/var/lib/kubelet/pods/2bd00201-829d-4766-b66d-321c3e4bc637/volumes"
Sep 17 17:53:49.330043 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:49.330019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:53:49.892592 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:49.892553    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:51.145591 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:53:51.145558    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:53:55.339311 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:53:55.339287    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:54:01.152714 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:54:01.152677    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:54:04.883716 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:54:04.883676    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d0bac_bc26_4e1c_bb7d_f0abd611392e.slice/crio-0cc53ef680eace33cc348a2166f934d2fe173e189ec1781a01a28915e8a29db5\": RecentStats: unable to find data in memory cache]"
Sep 17 17:54:25.339962 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:54:25.339908    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:54:35.339850 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:54:35.339771    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:54:45.340273 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:54:45.340225    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:54:55.340668 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:54:55.340632    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:54:57.272209 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:54:57.272164    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.23:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.23:8080: connect: connection refused"
Sep 17 17:55:07.852853 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:07.852815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:55:09.229031 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:09.228998    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"]
Sep 17 17:55:09.229420 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:09.229244    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container" containerID="cri-o://78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d" gracePeriod=30
Sep 17 17:55:09.229420 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:09.229270    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kube-rbac-proxy" containerID="cri-o://eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0" gracePeriod=30
Sep 17 17:55:09.602653 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:09.602616    2562 generic.go:356] "Generic (PLEG): container finished" podID="c491722a-62fc-4174-bbd7-99ca26917063" containerID="eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0" exitCode=2
Sep 17 17:55:09.602653 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:09.602653    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerDied","Data":"eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0"}
Sep 17 17:55:10.334610 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:10.334573    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Sep 17 17:55:14.269234 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.269208    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:55:14.407699 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407624    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"]
Sep 17 17:55:14.407893 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407881    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407895    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407905    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kube-rbac-proxy"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407925    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kube-rbac-proxy"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407935    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407941    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407950    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="storage-initializer"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407955    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="storage-initializer"
Sep 17 17:55:14.407958 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407960    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="storage-initializer"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407965    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="storage-initializer"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407977    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407982    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407988    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.407993    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.408046    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kserve-container"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.408055    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="agent"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.408061    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kserve-container"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.408066    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd00201-829d-4766-b66d-321c3e4bc637" containerName="kube-rbac-proxy"
Sep 17 17:55:14.408192 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.408073    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c491722a-62fc-4174-bbd7-99ca26917063" containerName="kube-rbac-proxy"
Sep 17 17:55:14.410954 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.410939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.413219 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.413195    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:55:14.413315 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.413248    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 17:55:14.423686 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.423665    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"]
Sep 17 17:55:14.441031 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441005    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls\") pod \"c491722a-62fc-4174-bbd7-99ca26917063\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") "
Sep 17 17:55:14.441149 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441044    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz7z4\" (UniqueName: \"kubernetes.io/projected/c491722a-62fc-4174-bbd7-99ca26917063-kube-api-access-gz7z4\") pod \"c491722a-62fc-4174-bbd7-99ca26917063\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") "
Sep 17 17:55:14.441149 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-cabundle-cert\") pod \"c491722a-62fc-4174-bbd7-99ca26917063\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") "
Sep 17 17:55:14.441149 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441114    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491722a-62fc-4174-bbd7-99ca26917063-kserve-provision-location\") pod \"c491722a-62fc-4174-bbd7-99ca26917063\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") "
Sep 17 17:55:14.441318 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441159    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-kube-rbac-proxy-sar-config\") pod \"c491722a-62fc-4174-bbd7-99ca26917063\" (UID: \"c491722a-62fc-4174-bbd7-99ca26917063\") "
Sep 17 17:55:14.441549 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441453    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c491722a-62fc-4174-bbd7-99ca26917063" (UID: "c491722a-62fc-4174-bbd7-99ca26917063"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:14.441549 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441488    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c491722a-62fc-4174-bbd7-99ca26917063-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c491722a-62fc-4174-bbd7-99ca26917063" (UID: "c491722a-62fc-4174-bbd7-99ca26917063"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:55:14.441802 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.441781    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c491722a-62fc-4174-bbd7-99ca26917063" (UID: "c491722a-62fc-4174-bbd7-99ca26917063"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:14.443253 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.443232    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c491722a-62fc-4174-bbd7-99ca26917063" (UID: "c491722a-62fc-4174-bbd7-99ca26917063"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:14.443361 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.443320    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c491722a-62fc-4174-bbd7-99ca26917063-kube-api-access-gz7z4" (OuterVolumeSpecName: "kube-api-access-gz7z4") pod "c491722a-62fc-4174-bbd7-99ca26917063" (UID: "c491722a-62fc-4174-bbd7-99ca26917063"). InnerVolumeSpecName "kube-api-access-gz7z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:14.541847 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.541816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvwc\" (UniqueName: \"kubernetes.io/projected/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-api-access-spvwc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.541991 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.541855    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.541991 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.541874    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.541991 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.541894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.541991 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.541963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.542137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.542005    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c491722a-62fc-4174-bbd7-99ca26917063-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:55:14.542137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.542015    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:55:14.542137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.542025    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c491722a-62fc-4174-bbd7-99ca26917063-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:55:14.542137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.542033    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7z4\" (UniqueName: \"kubernetes.io/projected/c491722a-62fc-4174-bbd7-99ca26917063-kube-api-access-gz7z4\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:55:14.542137 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.542042    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c491722a-62fc-4174-bbd7-99ca26917063-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:55:14.623266 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.623241    2562 generic.go:356] "Generic (PLEG): container finished" podID="c491722a-62fc-4174-bbd7-99ca26917063" containerID="78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d" exitCode=0
Sep 17 17:55:14.623372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.623284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerDied","Data":"78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d"}
Sep 17 17:55:14.623372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.623295    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"
Sep 17 17:55:14.623372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.623342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8" event={"ID":"c491722a-62fc-4174-bbd7-99ca26917063","Type":"ContainerDied","Data":"ead3157001f923705cb8de2850e571f78d2738e4e12a3905066bfb43ee967e86"}
Sep 17 17:55:14.623372 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.623368    2562 scope.go:117] "RemoveContainer" containerID="eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0"
Sep 17 17:55:14.631904 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.631882    2562 scope.go:117] "RemoveContainer" containerID="78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d"
Sep 17 17:55:14.638833 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.638818    2562 scope.go:117] "RemoveContainer" containerID="c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145"
Sep 17 17:55:14.643051 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.643026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvwc\" (UniqueName: \"kubernetes.io/projected/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-api-access-spvwc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.643846 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.643079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.643846 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.643115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.643846 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.643154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.643846 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.643190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.643846 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.643585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.644219 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.643883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.644345 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.644296    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.645335 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.645303    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"]
Sep 17 17:55:14.646007 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.645978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.646551 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.646525    2562 scope.go:117] "RemoveContainer" containerID="eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0"
Sep 17 17:55:14.646815 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:55:14.646798    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0\": container with ID starting with eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0 not found: ID does not exist" containerID="eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0"
Sep 17 17:55:14.646905 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.646829    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0"} err="failed to get container status \"eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0\": rpc error: code = NotFound desc = could not find container \"eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0\": container with ID starting with eb7cb5801dee8b34ed304e4679beb6e4bc21d0ecfb1f3e2f105d0cefab81b1c0 not found: ID does not exist"
Sep 17 17:55:14.646905 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.646845    2562 scope.go:117] "RemoveContainer" containerID="78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d"
Sep 17 17:55:14.647148 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:55:14.647064    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d\": container with ID starting with 78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d not found: ID does not exist" containerID="78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d"
Sep 17 17:55:14.647148 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.647093    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d"} err="failed to get container status \"78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d\": rpc error: code = NotFound desc = could not find container \"78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d\": container with ID starting with 78eaa0df466ddf4c339b10da63665eb68a6ab6fa519300d800b587a1b7a26a4d not found: ID does not exist"
Sep 17 17:55:14.647148 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.647117    2562 scope.go:117] "RemoveContainer" containerID="c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145"
Sep 17 17:55:14.647426 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:55:14.647403    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145\": container with ID starting with c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145 not found: ID does not exist" containerID="c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145"
Sep 17 17:55:14.647550 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.647436    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145"} err="failed to get container status \"c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145\": rpc error: code = NotFound desc = could not find container \"c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145\": container with ID starting with c088e300dc2698f00b322da10e101f31d11955e008bdbad0943a79478e405145 not found: ID does not exist"
Sep 17 17:55:14.649464 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.649443    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-x6qq8"]
Sep 17 17:55:14.652516 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.652494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvwc\" (UniqueName: \"kubernetes.io/projected/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-api-access-spvwc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.721343 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.721265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:14.990175 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:14.990140    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"]
Sep 17 17:55:14.992812 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:55:14.992789    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa2c46f_cb40_42e0_84ba_ca8ae7534de5.slice/crio-f932224e7b93a18e6107bfbf0294fc54221540b3d5e23a8669b80ee29ad9f462 WatchSource:0}: Error finding container f932224e7b93a18e6107bfbf0294fc54221540b3d5e23a8669b80ee29ad9f462: Status 404 returned error can't find the container with id f932224e7b93a18e6107bfbf0294fc54221540b3d5e23a8669b80ee29ad9f462
Sep 17 17:55:15.276367 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:15.276295    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c491722a-62fc-4174-bbd7-99ca26917063" path="/var/lib/kubelet/pods/c491722a-62fc-4174-bbd7-99ca26917063/volumes"
Sep 17 17:55:15.628487 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:15.628391    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerStarted","Data":"d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08"}
Sep 17 17:55:15.628487 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:15.628471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerStarted","Data":"f932224e7b93a18e6107bfbf0294fc54221540b3d5e23a8669b80ee29ad9f462"}
Sep 17 17:55:19.645585 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:19.645552    2562 generic.go:356] "Generic (PLEG): container finished" podID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerID="d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08" exitCode=0
Sep 17 17:55:19.646012 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:19.645625    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerDied","Data":"d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08"}
Sep 17 17:55:20.651605 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:20.651573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerStarted","Data":"9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874"}
Sep 17 17:55:20.652014 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:20.651644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerStarted","Data":"1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88"}
Sep 17 17:55:20.652014 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:20.651837    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:20.672963 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:20.672901    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podStartSLOduration=2.879548394 podStartE2EDuration="6.672887842s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793339448s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:20.671968879 +0000 UTC m=+1211.859404729" watchObservedRunningTime="2026-09-17 17:55:20.672887842 +0000 UTC m=+1211.860323690"
Sep 17 17:55:21.655427 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:21.655392    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:27.664996 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:27.664965    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:55:57.666769 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:55:57.666732    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:56:07.665743 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:07.665666    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:56:17.665575 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:17.665538    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:56:27.666415 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:27.666378    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:56:32.272014 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:32.271969    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.24:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 17:56:42.848293 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:42.848259    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:56:44.457397 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:44.457369    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"]
Sep 17 17:56:44.457838 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:44.457627    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container" containerID="cri-o://1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88" gracePeriod=30
Sep 17 17:56:44.457838 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:44.457689    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kube-rbac-proxy" containerID="cri-o://9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874" gracePeriod=30
Sep 17 17:56:44.941768 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:44.941737    2562 generic.go:356] "Generic (PLEG): container finished" podID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerID="9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874" exitCode=2
Sep 17 17:56:44.941952 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:44.941808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerDied","Data":"9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874"}
Sep 17 17:56:47.659929 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:47.659864    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 17:56:49.301902 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.301877    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:56:49.404635 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.404567    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-cabundle-cert\") pod \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") "
Sep 17 17:56:49.404770 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.404637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kserve-provision-location\") pod \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") "
Sep 17 17:56:49.404770 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.404674    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spvwc\" (UniqueName: \"kubernetes.io/projected/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-api-access-spvwc\") pod \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") "
Sep 17 17:56:49.404770 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.404723    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-rbac-proxy-sar-config\") pod \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") "
Sep 17 17:56:49.404770 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.404746    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-proxy-tls\") pod \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\" (UID: \"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5\") "
Sep 17 17:56:49.405000 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.404979    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" (UID: "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:49.405042 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.404956    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" (UID: "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:56:49.405157 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.405128    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" (UID: "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:56:49.406808 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.406780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" (UID: "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:56:49.406941 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.406882    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-api-access-spvwc" (OuterVolumeSpecName: "kube-api-access-spvwc") pod "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" (UID: "7aa2c46f-cb40-42e0-84ba-ca8ae7534de5"). InnerVolumeSpecName "kube-api-access-spvwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:56:49.505678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.505654    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:56:49.505678 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.505675    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:56:49.505829 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.505685    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:56:49.505829 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.505694    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:56:49.505829 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.505702    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvwc\" (UniqueName: \"kubernetes.io/projected/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5-kube-api-access-spvwc\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:56:49.663198 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663127    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"]
Sep 17 17:56:49.663390 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663378    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kube-rbac-proxy"
Sep 17 17:56:49.663431 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663391    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kube-rbac-proxy"
Sep 17 17:56:49.663431 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663402    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container"
Sep 17 17:56:49.663431 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663407    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container"
Sep 17 17:56:49.663431 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663419    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="storage-initializer"
Sep 17 17:56:49.663431 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663424    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="storage-initializer"
Sep 17 17:56:49.663581 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663466    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kserve-container"
Sep 17 17:56:49.663581 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.663474    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerName="kube-rbac-proxy"
Sep 17 17:56:49.666642 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.666625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.668985 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.668952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 17:56:49.669088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.669046    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:56:49.677453 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.677433    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"]
Sep 17 17:56:49.807992 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.807961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjq85\" (UniqueName: \"kubernetes.io/projected/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-api-access-hjq85\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.807992 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.807994    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.808179 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.808031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.808179 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.808099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db2885c-cbf1-4b45-80e6-6924a210f779-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.808179 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.808126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db2885c-cbf1-4b45-80e6-6924a210f779-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.909174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909140    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db2885c-cbf1-4b45-80e6-6924a210f779-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.909174 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db2885c-cbf1-4b45-80e6-6924a210f779-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.909383 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjq85\" (UniqueName: \"kubernetes.io/projected/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-api-access-hjq85\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.909383 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.909383 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.909620 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db2885c-cbf1-4b45-80e6-6924a210f779-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.909965 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.910102 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.909946    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.911784 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.911764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db2885c-cbf1-4b45-80e6-6924a210f779-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.917675 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.917629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjq85\" (UniqueName: \"kubernetes.io/projected/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-api-access-hjq85\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.961993 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.961966    2562 generic.go:356] "Generic (PLEG): container finished" podID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" containerID="1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88" exitCode=0
Sep 17 17:56:49.962085 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.962027    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"
Sep 17 17:56:49.962085 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.962053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerDied","Data":"1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88"}
Sep 17 17:56:49.962165 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.962113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts" event={"ID":"7aa2c46f-cb40-42e0-84ba-ca8ae7534de5","Type":"ContainerDied","Data":"f932224e7b93a18e6107bfbf0294fc54221540b3d5e23a8669b80ee29ad9f462"}
Sep 17 17:56:49.962165 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.962139    2562 scope.go:117] "RemoveContainer" containerID="9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874"
Sep 17 17:56:49.970009 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.969992    2562 scope.go:117] "RemoveContainer" containerID="1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88"
Sep 17 17:56:49.976623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.976602    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:49.977088 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.977067    2562 scope.go:117] "RemoveContainer" containerID="d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08"
Sep 17 17:56:49.983591 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.983569    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"]
Sep 17 17:56:49.985693 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.985681    2562 scope.go:117] "RemoveContainer" containerID="9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874"
Sep 17 17:56:49.985967 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:56:49.985949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874\": container with ID starting with 9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874 not found: ID does not exist" containerID="9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874"
Sep 17 17:56:49.986047 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.985980    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874"} err="failed to get container status \"9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874\": rpc error: code = NotFound desc = could not find container \"9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874\": container with ID starting with 9878c74d61215b2065d91022c67c9e4662257e710c6b41c955c178945b025874 not found: ID does not exist"
Sep 17 17:56:49.986047 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.986004    2562 scope.go:117] "RemoveContainer" containerID="1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88"
Sep 17 17:56:49.986252 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:56:49.986236    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88\": container with ID starting with 1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88 not found: ID does not exist" containerID="1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88"
Sep 17 17:56:49.986316 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.986259    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88"} err="failed to get container status \"1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88\": rpc error: code = NotFound desc = could not find container \"1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88\": container with ID starting with 1367ad5b29a13902cd9f59528c18f81072ce362015c34c7406ef112ad2f4ea88 not found: ID does not exist"
Sep 17 17:56:49.986316 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.986279    2562 scope.go:117] "RemoveContainer" containerID="d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08"
Sep 17 17:56:49.986510 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:56:49.986495    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08\": container with ID starting with d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08 not found: ID does not exist" containerID="d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08"
Sep 17 17:56:49.986567 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.986515    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08"} err="failed to get container status \"d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08\": rpc error: code = NotFound desc = could not find container \"d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08\": container with ID starting with d75140966129b670050d1b5620ee292520f355addf26990019f54a456f1f2c08 not found: ID does not exist"
Sep 17 17:56:49.989732 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:49.989711    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-pfnts"]
Sep 17 17:56:50.253049 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:50.251811    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"]
Sep 17 17:56:50.256198 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:56:50.256171    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db2885c_cbf1_4b45_80e6_6924a210f779.slice/crio-fb6c47316dc6e2480ac30c28bc565e065ee75cecdb83226da405a9dadda74044 WatchSource:0}: Error finding container fb6c47316dc6e2480ac30c28bc565e065ee75cecdb83226da405a9dadda74044: Status 404 returned error can't find the container with id fb6c47316dc6e2480ac30c28bc565e065ee75cecdb83226da405a9dadda74044
Sep 17 17:56:50.967127 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:50.967077    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerStarted","Data":"790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01"}
Sep 17 17:56:50.967544 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:50.967164    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerStarted","Data":"fb6c47316dc6e2480ac30c28bc565e065ee75cecdb83226da405a9dadda74044"}
Sep 17 17:56:51.275646 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:51.275613    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa2c46f-cb40-42e0-84ba-ca8ae7534de5" path="/var/lib/kubelet/pods/7aa2c46f-cb40-42e0-84ba-ca8ae7534de5/volumes"
Sep 17 17:56:54.982109 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:54.982076    2562 generic.go:356] "Generic (PLEG): container finished" podID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerID="790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01" exitCode=0
Sep 17 17:56:54.982473 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:54.982127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerDied","Data":"790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01"}
Sep 17 17:56:55.988235 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:55.988205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerStarted","Data":"d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87"}
Sep 17 17:56:55.988647 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:55.988264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerStarted","Data":"a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca"}
Sep 17 17:56:55.988647 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:55.988483    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:55.988718 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:55.988657    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:56:56.011032 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:56:56.010955    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podStartSLOduration=3.226799848 podStartE2EDuration="7.010938445s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.784138597s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:56:56.01059063 +0000 UTC m=+1307.198026479" watchObservedRunningTime="2026-09-17 17:56:56.010938445 +0000 UTC m=+1307.198374297"
Sep 17 17:57:01.997171 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:57:01.997139    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:57:31.998393 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:57:31.998312    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:57:41.998170 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:57:41.998128    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:57:51.998588 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:57:51.998552    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:58:01.998063 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:01.998028    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:58:02.271961 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:02.271858    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:58:12.756638 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:12.756610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:58:19.815525 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:19.815491    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"]
Sep 17 17:58:19.816069 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:19.815743    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container" containerID="cri-o://a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca" gracePeriod=30
Sep 17 17:58:19.816069 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:19.815770    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kube-rbac-proxy" containerID="cri-o://d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87" gracePeriod=30
Sep 17 17:58:20.286977 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:20.286942    2562 generic.go:356] "Generic (PLEG): container finished" podID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerID="d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87" exitCode=2
Sep 17 17:58:20.287143 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:20.287014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerDied","Data":"d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87"}
Sep 17 17:58:21.992317 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:21.992277    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 17:58:22.272268 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:22.272177    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.25:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 17:58:24.547425 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.547402    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:58:24.666488 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666414    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-cabundle-cert\") pod \"6db2885c-cbf1-4b45-80e6-6924a210f779\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") "
Sep 17 17:58:24.666488 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666467    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjq85\" (UniqueName: \"kubernetes.io/projected/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-api-access-hjq85\") pod \"6db2885c-cbf1-4b45-80e6-6924a210f779\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") "
Sep 17 17:58:24.666702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666498    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db2885c-cbf1-4b45-80e6-6924a210f779-kserve-provision-location\") pod \"6db2885c-cbf1-4b45-80e6-6924a210f779\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") "
Sep 17 17:58:24.666702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666534    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db2885c-cbf1-4b45-80e6-6924a210f779-proxy-tls\") pod \"6db2885c-cbf1-4b45-80e6-6924a210f779\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") "
Sep 17 17:58:24.666702 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-rbac-proxy-sar-config\") pod \"6db2885c-cbf1-4b45-80e6-6924a210f779\" (UID: \"6db2885c-cbf1-4b45-80e6-6924a210f779\") "
Sep 17 17:58:24.666955 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666896    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db2885c-cbf1-4b45-80e6-6924a210f779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6db2885c-cbf1-4b45-80e6-6924a210f779" (UID: "6db2885c-cbf1-4b45-80e6-6924a210f779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:58:24.666955 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666935    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6db2885c-cbf1-4b45-80e6-6924a210f779" (UID: "6db2885c-cbf1-4b45-80e6-6924a210f779"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:24.667101 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.666961    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6db2885c-cbf1-4b45-80e6-6924a210f779" (UID: "6db2885c-cbf1-4b45-80e6-6924a210f779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:58:24.668594 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.668567    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-api-access-hjq85" (OuterVolumeSpecName: "kube-api-access-hjq85") pod "6db2885c-cbf1-4b45-80e6-6924a210f779" (UID: "6db2885c-cbf1-4b45-80e6-6924a210f779"). InnerVolumeSpecName "kube-api-access-hjq85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:58:24.668594 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.668583    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db2885c-cbf1-4b45-80e6-6924a210f779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6db2885c-cbf1-4b45-80e6-6924a210f779" (UID: "6db2885c-cbf1-4b45-80e6-6924a210f779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:58:24.767031 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.767000    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6db2885c-cbf1-4b45-80e6-6924a210f779-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:58:24.767031 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.767022    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:58:24.767031 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.767031    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6db2885c-cbf1-4b45-80e6-6924a210f779-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:58:24.767247 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.767040    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjq85\" (UniqueName: \"kubernetes.io/projected/6db2885c-cbf1-4b45-80e6-6924a210f779-kube-api-access-hjq85\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:58:24.767247 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:24.767048    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6db2885c-cbf1-4b45-80e6-6924a210f779-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 17:58:25.304378 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.304346    2562 generic.go:356] "Generic (PLEG): container finished" podID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerID="a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca" exitCode=0
Sep 17 17:58:25.304529 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.304396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerDied","Data":"a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca"}
Sep 17 17:58:25.304529 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.304408    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"
Sep 17 17:58:25.304529 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.304446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj" event={"ID":"6db2885c-cbf1-4b45-80e6-6924a210f779","Type":"ContainerDied","Data":"fb6c47316dc6e2480ac30c28bc565e065ee75cecdb83226da405a9dadda74044"}
Sep 17 17:58:25.304529 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.304474    2562 scope.go:117] "RemoveContainer" containerID="d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87"
Sep 17 17:58:25.317466 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.317444    2562 scope.go:117] "RemoveContainer" containerID="a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca"
Sep 17 17:58:25.323822 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.323800    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"]
Sep 17 17:58:25.324774 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.324758    2562 scope.go:117] "RemoveContainer" containerID="790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01"
Sep 17 17:58:25.328392 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.328372    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-dz9zj"]
Sep 17 17:58:25.331754 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.331736    2562 scope.go:117] "RemoveContainer" containerID="d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87"
Sep 17 17:58:25.332041 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:58:25.332021    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87\": container with ID starting with d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87 not found: ID does not exist" containerID="d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87"
Sep 17 17:58:25.332114 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.332049    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87"} err="failed to get container status \"d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87\": rpc error: code = NotFound desc = could not find container \"d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87\": container with ID starting with d0c9cccbc4c2da507c226860e666e466ec9c7a3d144976204b86866d6eccbb87 not found: ID does not exist"
Sep 17 17:58:25.332114 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.332065    2562 scope.go:117] "RemoveContainer" containerID="a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca"
Sep 17 17:58:25.332312 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:58:25.332296    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca\": container with ID starting with a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca not found: ID does not exist" containerID="a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca"
Sep 17 17:58:25.332352 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.332318    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca"} err="failed to get container status \"a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca\": rpc error: code = NotFound desc = could not find container \"a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca\": container with ID starting with a4fa98ff9371e769203f16187acd7797005083ab166d032779f2e6d821e62dca not found: ID does not exist"
Sep 17 17:58:25.332352 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.332333    2562 scope.go:117] "RemoveContainer" containerID="790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01"
Sep 17 17:58:25.332565 ip-10-0-132-143 kubenswrapper[2562]: E0917 17:58:25.332545    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01\": container with ID starting with 790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01 not found: ID does not exist" containerID="790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01"
Sep 17 17:58:25.332617 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:25.332569    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01"} err="failed to get container status \"790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01\": rpc error: code = NotFound desc = could not find container \"790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01\": container with ID starting with 790e0f7cae25ccf6248119aeff78cdc4211a4a3e053bec4a08d668de3752ff01 not found: ID does not exist"
Sep 17 17:58:27.276073 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:27.276046    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" path="/var/lib/kubelet/pods/6db2885c-cbf1-4b45-80e6-6924a210f779/volumes"
Sep 17 17:58:31.893572 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.893539    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"]
Sep 17 17:58:31.898244 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898212    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="storage-initializer"
Sep 17 17:58:31.898379 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898346    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="storage-initializer"
Sep 17 17:58:31.898379 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898368    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kube-rbac-proxy"
Sep 17 17:58:31.898379 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898377    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kube-rbac-proxy"
Sep 17 17:58:31.898583 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898393    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container"
Sep 17 17:58:31.898583 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898401    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container"
Sep 17 17:58:31.898583 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898498    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kube-rbac-proxy"
Sep 17 17:58:31.898744 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.898673    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db2885c-cbf1-4b45-80e6-6924a210f779" containerName="kserve-container"
Sep 17 17:58:31.903243 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.903221    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:31.906050 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.906026    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 17:58:31.906150 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.906062    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 17:58:31.906150 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.906088    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:58:31.906150 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.906036    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 17:58:31.906150 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.906036    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:58:31.906341 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.906036    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:58:31.906603 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.906590    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:58:31.909531 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:31.909511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"]
Sep 17 17:58:32.016461 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.016431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26nc\" (UniqueName: \"kubernetes.io/projected/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-api-access-k26nc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.016461 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.016464    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.016643 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.016489    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394e7a02-2d69-42f7-abd7-c45f140230f8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.016643 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.016560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.016643 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.016595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394e7a02-2d69-42f7-abd7-c45f140230f8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117073 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117204 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117082    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394e7a02-2d69-42f7-abd7-c45f140230f8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117204 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117303 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394e7a02-2d69-42f7-abd7-c45f140230f8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117365 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26nc\" (UniqueName: \"kubernetes.io/projected/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-api-access-k26nc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117496 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117477    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394e7a02-2d69-42f7-abd7-c45f140230f8-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117752 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.117752 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.117746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.119612 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.119591    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394e7a02-2d69-42f7-abd7-c45f140230f8-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.126506 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.126488    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26nc\" (UniqueName: \"kubernetes.io/projected/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-api-access-k26nc\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.214143 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.214081    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 17:58:32.494384 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:32.494352    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"]
Sep 17 17:58:32.498018 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:58:32.497983    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394e7a02_2d69_42f7_abd7_c45f140230f8.slice/crio-4bcfb6162f2b5975ffeeff99d8c367c03fa798ce30b7c46aa528dd00d1e60ef5 WatchSource:0}: Error finding container 4bcfb6162f2b5975ffeeff99d8c367c03fa798ce30b7c46aa528dd00d1e60ef5: Status 404 returned error can't find the container with id 4bcfb6162f2b5975ffeeff99d8c367c03fa798ce30b7c46aa528dd00d1e60ef5
Sep 17 17:58:33.333173 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:33.333143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerStarted","Data":"64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8"}
Sep 17 17:58:33.333543 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:33.333196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerStarted","Data":"4bcfb6162f2b5975ffeeff99d8c367c03fa798ce30b7c46aa528dd00d1e60ef5"}
Sep 17 17:58:59.259550 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.259479    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"]
Sep 17 17:58:59.264261 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.264243    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.267038 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.267021    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 17:58:59.267150 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.267021    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 17:58:59.276335 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.276313    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"]
Sep 17 17:58:59.399620 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.399582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.399789 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.399627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35e3dca4-62f1-4069-844a-d7fa654bdd9e-proxy-tls\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.399789 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.399660    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.399789 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.399685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76zt\" (UniqueName: \"kubernetes.io/projected/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-api-access-n76zt\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.399789 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.399705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-cabundle-cert\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.500631 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.500608    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.500785 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.500636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35e3dca4-62f1-4069-844a-d7fa654bdd9e-proxy-tls\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.500785 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.500658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.500785 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.500690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76zt\" (UniqueName: \"kubernetes.io/projected/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-api-access-n76zt\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.500785 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.500713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-cabundle-cert\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.501413 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.501391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kserve-provision-location\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.501466 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.501398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-cabundle-cert\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.501466 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.501440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.503111 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.503088    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35e3dca4-62f1-4069-844a-d7fa654bdd9e-proxy-tls\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.510028 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.509977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76zt\" (UniqueName: \"kubernetes.io/projected/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-api-access-n76zt\") pod \"isvc-sklearn-predictor-78b4bb59c9-mh49z\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.575188 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.575161    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:58:59.853526 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:58:59.853431    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"]
Sep 17 17:58:59.856840 ip-10-0-132-143 kubenswrapper[2562]: W0917 17:58:59.856801    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e3dca4_62f1_4069_844a_d7fa654bdd9e.slice/crio-4ed74a8b6bd7f65139b0e0f0c7c90b2c62d160bdbf7686c7264f425cb4eeb2d0 WatchSource:0}: Error finding container 4ed74a8b6bd7f65139b0e0f0c7c90b2c62d160bdbf7686c7264f425cb4eeb2d0: Status 404 returned error can't find the container with id 4ed74a8b6bd7f65139b0e0f0c7c90b2c62d160bdbf7686c7264f425cb4eeb2d0
Sep 17 17:59:00.427039 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:00.426996    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerStarted","Data":"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"}
Sep 17 17:59:00.427428 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:00.427076    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerStarted","Data":"4ed74a8b6bd7f65139b0e0f0c7c90b2c62d160bdbf7686c7264f425cb4eeb2d0"}
Sep 17 17:59:04.445147 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:04.445113    2562 generic.go:356] "Generic (PLEG): container finished" podID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerID="ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87" exitCode=0
Sep 17 17:59:04.445546 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:04.445188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerDied","Data":"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"}
Sep 17 17:59:04.446793 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:04.446773    2562 generic.go:356] "Generic (PLEG): container finished" podID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerID="64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8" exitCode=0
Sep 17 17:59:04.446888 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:04.446823    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerDied","Data":"64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8"}
Sep 17 17:59:04.448083 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:04.448058    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:59:05.455214 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:05.455178    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerStarted","Data":"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"}
Sep 17 17:59:05.455646 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:05.455250    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerStarted","Data":"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"}
Sep 17 17:59:05.455646 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:05.455527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:59:06.461346 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:06.460947    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:59:06.462224 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:06.462189    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 17:59:07.465067 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:07.465023    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 17:59:12.470097 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:12.470039    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 17:59:12.470623 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:12.470595    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 17:59:12.492866 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:12.492674    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podStartSLOduration=9.809314004 podStartE2EDuration="13.492657008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683343004s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:05.477023733 +0000 UTC m=+1436.664459587" watchObservedRunningTime="2026-09-17 17:59:12.492657008 +0000 UTC m=+1443.680092858"
Sep 17 17:59:22.471467 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:22.471426    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 17:59:32.471301 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:32.471262    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 17:59:42.470775 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:42.470736    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 17:59:52.470670 ip-10-0-132-143 kubenswrapper[2562]: I0917 17:59:52.470632    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:00:02.471317 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:02.471274    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:00:12.471594 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:12.471555    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:00:22.471168 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:22.471114    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:00:32.472186 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:32.472063    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 18:00:39.370610 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:39.370577    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"]
Sep 17 18:00:39.371186 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:39.370827    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" containerID="cri-o://ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c" gracePeriod=30
Sep 17 18:00:39.371186 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:39.370868    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kube-rbac-proxy" containerID="cri-o://eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210" gracePeriod=30
Sep 17 18:00:42.465605 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:42.465552    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:00:42.471070 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:42.471040    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:00:47.465835 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:47.465787    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:00:52.465389 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:52.465346    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 18:00:52.465945 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:52.465492    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 18:00:52.470746 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:52.470717    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 18:00:55.869253 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.869228    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 18:00:55.907307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.907245    2562 generic.go:356] "Generic (PLEG): container finished" podID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerID="eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210" exitCode=2
Sep 17 18:00:55.907307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.907269    2562 generic.go:356] "Generic (PLEG): container finished" podID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerID="ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c" exitCode=0
Sep 17 18:00:55.907307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.907298    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"
Sep 17 18:00:55.907522 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.907322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerDied","Data":"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"}
Sep 17 18:00:55.907522 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.907384    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerDied","Data":"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"}
Sep 17 18:00:55.907522 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.907407    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z" event={"ID":"35e3dca4-62f1-4069-844a-d7fa654bdd9e","Type":"ContainerDied","Data":"4ed74a8b6bd7f65139b0e0f0c7c90b2c62d160bdbf7686c7264f425cb4eeb2d0"}
Sep 17 18:00:55.907522 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.907433    2562 scope.go:117] "RemoveContainer" containerID="eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"
Sep 17 18:00:55.949068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.949047    2562 scope.go:117] "RemoveContainer" containerID="ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"
Sep 17 18:00:55.958714 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.958691    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n76zt\" (UniqueName: \"kubernetes.io/projected/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-api-access-n76zt\") pod \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") "
Sep 17 18:00:55.958807 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.958738    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-rbac-proxy-sar-config\") pod \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") "
Sep 17 18:00:55.958807 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.958766    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35e3dca4-62f1-4069-844a-d7fa654bdd9e-proxy-tls\") pod \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") "
Sep 17 18:00:55.958901 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.958812    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kserve-provision-location\") pod \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") "
Sep 17 18:00:55.958901 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.958865    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-cabundle-cert\") pod \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\" (UID: \"35e3dca4-62f1-4069-844a-d7fa654bdd9e\") "
Sep 17 18:00:55.959233 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.959200    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35e3dca4-62f1-4069-844a-d7fa654bdd9e" (UID: "35e3dca4-62f1-4069-844a-d7fa654bdd9e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:00:55.959347 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.959230    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "35e3dca4-62f1-4069-844a-d7fa654bdd9e" (UID: "35e3dca4-62f1-4069-844a-d7fa654bdd9e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:55.959347 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.959335    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35e3dca4-62f1-4069-844a-d7fa654bdd9e" (UID: "35e3dca4-62f1-4069-844a-d7fa654bdd9e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:00:55.960727 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.960702    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e3dca4-62f1-4069-844a-d7fa654bdd9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35e3dca4-62f1-4069-844a-d7fa654bdd9e" (UID: "35e3dca4-62f1-4069-844a-d7fa654bdd9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:00:55.960901 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:55.960882    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-api-access-n76zt" (OuterVolumeSpecName: "kube-api-access-n76zt") pod "35e3dca4-62f1-4069-844a-d7fa654bdd9e" (UID: "35e3dca4-62f1-4069-844a-d7fa654bdd9e"). InnerVolumeSpecName "kube-api-access-n76zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:00:56.060341 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.060313    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n76zt\" (UniqueName: \"kubernetes.io/projected/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-api-access-n76zt\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:00:56.060341 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.060340    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:00:56.060341 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.060349    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35e3dca4-62f1-4069-844a-d7fa654bdd9e-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:00:56.060607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.060358    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35e3dca4-62f1-4069-844a-d7fa654bdd9e-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:00:56.060607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.060368    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35e3dca4-62f1-4069-844a-d7fa654bdd9e-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:00:56.111323 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.111302    2562 scope.go:117] "RemoveContainer" containerID="ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"
Sep 17 18:00:56.121151 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.121131    2562 scope.go:117] "RemoveContainer" containerID="eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"
Sep 17 18:00:56.121404 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:00:56.121381    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210\": container with ID starting with eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210 not found: ID does not exist" containerID="eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"
Sep 17 18:00:56.121498 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.121410    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"} err="failed to get container status \"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210\": rpc error: code = NotFound desc = could not find container \"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210\": container with ID starting with eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210 not found: ID does not exist"
Sep 17 18:00:56.121498 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.121428    2562 scope.go:117] "RemoveContainer" containerID="ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"
Sep 17 18:00:56.121664 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:00:56.121645    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c\": container with ID starting with ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c not found: ID does not exist" containerID="ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"
Sep 17 18:00:56.121721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.121669    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"} err="failed to get container status \"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c\": rpc error: code = NotFound desc = could not find container \"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c\": container with ID starting with ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c not found: ID does not exist"
Sep 17 18:00:56.121721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.121689    2562 scope.go:117] "RemoveContainer" containerID="ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"
Sep 17 18:00:56.121972 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:00:56.121901    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87\": container with ID starting with ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87 not found: ID does not exist" containerID="ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"
Sep 17 18:00:56.121972 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.121940    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"} err="failed to get container status \"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87\": rpc error: code = NotFound desc = could not find container \"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87\": container with ID starting with ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87 not found: ID does not exist"
Sep 17 18:00:56.121972 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.121955    2562 scope.go:117] "RemoveContainer" containerID="eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"
Sep 17 18:00:56.122254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.122232    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210"} err="failed to get container status \"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210\": rpc error: code = NotFound desc = could not find container \"eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210\": container with ID starting with eb8e14379985145cab05e264f41f495a6c6eaa5b13e53486ff5eb2900abce210 not found: ID does not exist"
Sep 17 18:00:56.122340 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.122258    2562 scope.go:117] "RemoveContainer" containerID="ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"
Sep 17 18:00:56.122537 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.122509    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c"} err="failed to get container status \"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c\": rpc error: code = NotFound desc = could not find container \"ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c\": container with ID starting with ba123bf128c54ccb8c16f2172e2bf6261cf66e653faa78e008a18143db0b559c not found: ID does not exist"
Sep 17 18:00:56.122640 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.122538    2562 scope.go:117] "RemoveContainer" containerID="ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"
Sep 17 18:00:56.122791 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.122763    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87"} err="failed to get container status \"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87\": rpc error: code = NotFound desc = could not find container \"ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87\": container with ID starting with ddcd13668f1a999267edffcac80777eaf99591fe0c7bef7d51c2df401ae01d87 not found: ID does not exist"
Sep 17 18:00:56.230651 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.230623    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"]
Sep 17 18:00:56.235002 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:56.234982    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-78b4bb59c9-mh49z"]
Sep 17 18:00:57.277254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:57.277222    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" path="/var/lib/kubelet/pods/35e3dca4-62f1-4069-844a-d7fa654bdd9e/volumes"
Sep 17 18:00:57.917054 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:57.917021    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerStarted","Data":"c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4"}
Sep 17 18:00:58.005377 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:57.917076    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerStarted","Data":"057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570"}
Sep 17 18:00:58.005377 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:57.917175    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 18:00:58.005377 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:57.938004    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" podStartSLOduration=3.236488213 podStartE2EDuration="2m26.937989473s" totalImagesPullingTime="1m52.369081275s" totalInitContainerRuntime="31.332419985s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:57.93697172 +0000 UTC m=+1549.124407569" watchObservedRunningTime="2026-09-17 18:00:57.937989473 +0000 UTC m=+1549.125425322"
Sep 17 18:00:58.920944 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:00:58.920908    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 18:01:04.930797 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:01:04.930772    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 18:02:30.304713 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:30.304636    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 18:02:38.492794 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:38.492764    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"]
Sep 17 18:02:38.493241 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:38.493028    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kserve-container" containerID="cri-o://057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570" gracePeriod=30
Sep 17 18:02:38.493241 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:38.493061    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kube-rbac-proxy" containerID="cri-o://c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4" gracePeriod=30
Sep 17 18:02:39.280512 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.280480    2562 generic.go:356] "Generic (PLEG): container finished" podID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerID="c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4" exitCode=2
Sep 17 18:02:39.280681 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.280558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerDied","Data":"c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4"}
Sep 17 18:02:39.826555 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.826534    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 18:02:39.880173 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.880107    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k26nc\" (UniqueName: \"kubernetes.io/projected/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-api-access-k26nc\") pod \"394e7a02-2d69-42f7-abd7-c45f140230f8\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") "
Sep 17 18:02:39.880173 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.880161    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394e7a02-2d69-42f7-abd7-c45f140230f8-proxy-tls\") pod \"394e7a02-2d69-42f7-abd7-c45f140230f8\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") "
Sep 17 18:02:39.880329 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.880199    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-rbac-proxy-sar-config\") pod \"394e7a02-2d69-42f7-abd7-c45f140230f8\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") "
Sep 17 18:02:39.880329 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.880221    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-cabundle-cert\") pod \"394e7a02-2d69-42f7-abd7-c45f140230f8\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") "
Sep 17 18:02:39.880329 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.880249    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394e7a02-2d69-42f7-abd7-c45f140230f8-kserve-provision-location\") pod \"394e7a02-2d69-42f7-abd7-c45f140230f8\" (UID: \"394e7a02-2d69-42f7-abd7-c45f140230f8\") "
Sep 17 18:02:39.880606 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.880580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "394e7a02-2d69-42f7-abd7-c45f140230f8" (UID: "394e7a02-2d69-42f7-abd7-c45f140230f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:39.880606 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.880598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "394e7a02-2d69-42f7-abd7-c45f140230f8" (UID: "394e7a02-2d69-42f7-abd7-c45f140230f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:39.882215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.882190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-api-access-k26nc" (OuterVolumeSpecName: "kube-api-access-k26nc") pod "394e7a02-2d69-42f7-abd7-c45f140230f8" (UID: "394e7a02-2d69-42f7-abd7-c45f140230f8"). InnerVolumeSpecName "kube-api-access-k26nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:39.882281 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.882237    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394e7a02-2d69-42f7-abd7-c45f140230f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "394e7a02-2d69-42f7-abd7-c45f140230f8" (UID: "394e7a02-2d69-42f7-abd7-c45f140230f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:39.884825 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.884804    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394e7a02-2d69-42f7-abd7-c45f140230f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "394e7a02-2d69-42f7-abd7-c45f140230f8" (UID: "394e7a02-2d69-42f7-abd7-c45f140230f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:39.981119 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.981084    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/394e7a02-2d69-42f7-abd7-c45f140230f8-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.981119 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.981117    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26nc\" (UniqueName: \"kubernetes.io/projected/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-api-access-k26nc\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.981119 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.981127    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/394e7a02-2d69-42f7-abd7-c45f140230f8-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.981311 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.981135    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:02:39.981311 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:39.981144    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/394e7a02-2d69-42f7-abd7-c45f140230f8-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:02:40.285729 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.285694    2562 generic.go:356] "Generic (PLEG): container finished" podID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerID="057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570" exitCode=0
Sep 17 18:02:40.285885 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.285764    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"
Sep 17 18:02:40.285885 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.285769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerDied","Data":"057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570"}
Sep 17 18:02:40.285885 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.285819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll" event={"ID":"394e7a02-2d69-42f7-abd7-c45f140230f8","Type":"ContainerDied","Data":"4bcfb6162f2b5975ffeeff99d8c367c03fa798ce30b7c46aa528dd00d1e60ef5"}
Sep 17 18:02:40.285885 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.285849    2562 scope.go:117] "RemoveContainer" containerID="c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4"
Sep 17 18:02:40.293905 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.293890    2562 scope.go:117] "RemoveContainer" containerID="057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570"
Sep 17 18:02:40.301167 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.301153    2562 scope.go:117] "RemoveContainer" containerID="64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8"
Sep 17 18:02:40.308680 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.308664    2562 scope.go:117] "RemoveContainer" containerID="c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4"
Sep 17 18:02:40.308758 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.308743    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"]
Sep 17 18:02:40.308966 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:02:40.308937    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4\": container with ID starting with c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4 not found: ID does not exist" containerID="c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4"
Sep 17 18:02:40.309068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.308966    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4"} err="failed to get container status \"c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4\": rpc error: code = NotFound desc = could not find container \"c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4\": container with ID starting with c492b20b6acd84a59792c71d17f596e7f51d840fd133eae16aff6b729975caf4 not found: ID does not exist"
Sep 17 18:02:40.309068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.308983    2562 scope.go:117] "RemoveContainer" containerID="057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570"
Sep 17 18:02:40.309256 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:02:40.309237    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570\": container with ID starting with 057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570 not found: ID does not exist" containerID="057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570"
Sep 17 18:02:40.309297 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.309263    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570"} err="failed to get container status \"057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570\": rpc error: code = NotFound desc = could not find container \"057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570\": container with ID starting with 057804b13b048a1ca2229c8aa168f0279c7636df8ebabd0309b6e7eec106f570 not found: ID does not exist"
Sep 17 18:02:40.309297 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.309286    2562 scope.go:117] "RemoveContainer" containerID="64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8"
Sep 17 18:02:40.309515 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:02:40.309500    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8\": container with ID starting with 64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8 not found: ID does not exist" containerID="64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8"
Sep 17 18:02:40.309554 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.309518    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8"} err="failed to get container status \"64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8\": rpc error: code = NotFound desc = could not find container \"64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8\": container with ID starting with 64d775862b836e86383310e0e1b1cd7e2071f4cb1053bbfa4ec5772777d877e8 not found: ID does not exist"
Sep 17 18:02:40.311810 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:40.311789    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-kkrll"]
Sep 17 18:02:41.276255 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:41.276223    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" path="/var/lib/kubelet/pods/394e7a02-2d69-42f7-abd7-c45f140230f8/volumes"
Sep 17 18:02:43.657668 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657636    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"]
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657871    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657881    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657892    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="storage-initializer"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657898    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="storage-initializer"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657903    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="storage-initializer"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657923    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="storage-initializer"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657931    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kserve-container"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657938    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kserve-container"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657946    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kube-rbac-proxy"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657951    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kube-rbac-proxy"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657956    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kube-rbac-proxy"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.657961    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kube-rbac-proxy"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.658006    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kserve-container"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.658013    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kube-rbac-proxy"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.658021    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="35e3dca4-62f1-4069-844a-d7fa654bdd9e" containerName="kserve-container"
Sep 17 18:02:43.658043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.658026    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="394e7a02-2d69-42f7-abd7-c45f140230f8" containerName="kube-rbac-proxy"
Sep 17 18:02:43.662753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.662734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.665528 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.665509    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 18:02:43.665703 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.665688    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:02:43.665778 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.665689    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:02:43.665778 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.665715    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:02:43.666073 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.666059    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:02:43.666399 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.666385    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:02:43.666470 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.666451    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:02:43.674818 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.674798    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"]
Sep 17 18:02:43.805842 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.805811    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnsn\" (UniqueName: \"kubernetes.io/projected/4a460157-321e-4514-a15c-a55e23ddb707-kube-api-access-5gnsn\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.805842 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.805848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.806052 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.805870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a460157-321e-4514-a15c-a55e23ddb707-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.806052 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.805973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.806052 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.806013    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a460157-321e-4514-a15c-a55e23ddb707-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.907151 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.907120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a460157-321e-4514-a15c-a55e23ddb707-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.907300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.907169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnsn\" (UniqueName: \"kubernetes.io/projected/4a460157-321e-4514-a15c-a55e23ddb707-kube-api-access-5gnsn\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.907300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.907204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.907300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.907228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a460157-321e-4514-a15c-a55e23ddb707-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.907300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.907251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.907766 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.907695    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a460157-321e-4514-a15c-a55e23ddb707-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.907989 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.907966    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.908081 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.908053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.909750 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.909727    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a460157-321e-4514-a15c-a55e23ddb707-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.916494 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.916474    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnsn\" (UniqueName: \"kubernetes.io/projected/4a460157-321e-4514-a15c-a55e23ddb707-kube-api-access-5gnsn\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:43.973114 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:43.973087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:02:44.244414 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:44.244390    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"]
Sep 17 18:02:44.246939 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:02:44.246897    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a460157_321e_4514_a15c_a55e23ddb707.slice/crio-41d7f6e38e786427880a5be2ff701fe9a52a29b9d00ce6c83c945b41356d10cb WatchSource:0}: Error finding container 41d7f6e38e786427880a5be2ff701fe9a52a29b9d00ce6c83c945b41356d10cb: Status 404 returned error can't find the container with id 41d7f6e38e786427880a5be2ff701fe9a52a29b9d00ce6c83c945b41356d10cb
Sep 17 18:02:44.300853 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:44.300819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerStarted","Data":"41d7f6e38e786427880a5be2ff701fe9a52a29b9d00ce6c83c945b41356d10cb"}
Sep 17 18:02:45.305550 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:02:45.305506    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerStarted","Data":"ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d"}
Sep 17 18:03:15.409412 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:15.409376    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a460157-321e-4514-a15c-a55e23ddb707" containerID="ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d" exitCode=0
Sep 17 18:03:15.409806 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:15.409452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerDied","Data":"ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d"}
Sep 17 18:03:16.416634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:16.416602    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerStarted","Data":"90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26"}
Sep 17 18:03:16.417149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:16.416659    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerStarted","Data":"aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774"}
Sep 17 18:03:16.417149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:16.416967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:03:16.417149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:16.417026    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:03:16.438344 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:16.438290    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" podStartSLOduration=3.220725157 podStartE2EDuration="33.438270271s" totalImagesPullingTime="0s" totalInitContainerRuntime="30.217545114s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:16.437286297 +0000 UTC m=+1687.624722160" watchObservedRunningTime="2026-09-17 18:03:16.438270271 +0000 UTC m=+1687.625706121"
Sep 17 18:03:22.425946 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:22.425902    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:03:49.912885 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.912855    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"]
Sep 17 18:03:49.916817 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.916800    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:49.919188 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.919162    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 18:03:49.919315 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.919288    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:03:49.928786 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.928764    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"]
Sep 17 18:03:49.959050 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.959023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:49.959163 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.959066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a717cc-199d-456d-bf27-90063f9970e6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:49.959163 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.959089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a717cc-199d-456d-bf27-90063f9970e6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:49.959163 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.959113    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:49.959288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:49.959165    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsb77\" (UniqueName: \"kubernetes.io/projected/a8a717cc-199d-456d-bf27-90063f9970e6-kube-api-access-nsb77\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.059775 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.059744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a717cc-199d-456d-bf27-90063f9970e6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.059775 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.059779    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a717cc-199d-456d-bf27-90063f9970e6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.060023 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.059797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.060023 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.059820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsb77\" (UniqueName: \"kubernetes.io/projected/a8a717cc-199d-456d-bf27-90063f9970e6-kube-api-access-nsb77\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.060023 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.059857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.060204 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.060183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a717cc-199d-456d-bf27-90063f9970e6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.060553 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.060531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.060633 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.060603    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.062312 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.062294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a717cc-199d-456d-bf27-90063f9970e6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.069197 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.069179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsb77\" (UniqueName: \"kubernetes.io/projected/a8a717cc-199d-456d-bf27-90063f9970e6-kube-api-access-nsb77\") pod \"isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.226772 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.226690    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:50.499787 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.499757    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"]
Sep 17 18:03:50.502498 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:03:50.502465    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a717cc_199d_456d_bf27_90063f9970e6.slice/crio-d485a0872375e79fc0948432128973c2a034555db3268c86aadc10fa7b7995f5 WatchSource:0}: Error finding container d485a0872375e79fc0948432128973c2a034555db3268c86aadc10fa7b7995f5: Status 404 returned error can't find the container with id d485a0872375e79fc0948432128973c2a034555db3268c86aadc10fa7b7995f5
Sep 17 18:03:50.534560 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:50.534532    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerStarted","Data":"d485a0872375e79fc0948432128973c2a034555db3268c86aadc10fa7b7995f5"}
Sep 17 18:03:51.539488 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:51.539452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerStarted","Data":"d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19"}
Sep 17 18:03:56.558373 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:56.558341    2562 generic.go:356] "Generic (PLEG): container finished" podID="a8a717cc-199d-456d-bf27-90063f9970e6" containerID="d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19" exitCode=0
Sep 17 18:03:56.558858 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:56.558382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerDied","Data":"d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19"}
Sep 17 18:03:57.565237 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:57.565203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerStarted","Data":"7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8"}
Sep 17 18:03:57.565671 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:57.565261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerStarted","Data":"ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea"}
Sep 17 18:03:57.565671 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:57.565547    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:57.565671 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:57.565604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:03:57.567079 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:57.567053    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:03:57.590721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:57.590675    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" podStartSLOduration=2.751692266 podStartE2EDuration="8.590662604s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.838970338s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:57.589312132 +0000 UTC m=+1728.776747982" watchObservedRunningTime="2026-09-17 18:03:57.590662604 +0000 UTC m=+1728.778098453"
Sep 17 18:03:58.568889 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:03:58.568841    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:04:03.573923 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:03.573889    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:04:03.574419 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:03.574394    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 18:04:13.574767 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:13.574726    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:04:21.758683 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:21.758653    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"]
Sep 17 18:04:21.759116 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:21.758901    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kserve-container" containerID="cri-o://ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea" gracePeriod=30
Sep 17 18:04:21.759116 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:21.758964    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kube-rbac-proxy" containerID="cri-o://7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8" gracePeriod=30
Sep 17 18:04:22.664201 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.664170    2562 generic.go:356] "Generic (PLEG): container finished" podID="a8a717cc-199d-456d-bf27-90063f9970e6" containerID="7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8" exitCode=2
Sep 17 18:04:22.664341 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.664244    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerDied","Data":"7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8"}
Sep 17 18:04:22.795396 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.795369    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:04:22.904171 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.904080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-cabundle-cert\") pod \"a8a717cc-199d-456d-bf27-90063f9970e6\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") "
Sep 17 18:04:22.904171 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.904124    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a717cc-199d-456d-bf27-90063f9970e6-kserve-provision-location\") pod \"a8a717cc-199d-456d-bf27-90063f9970e6\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") "
Sep 17 18:04:22.904171 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.904149    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a717cc-199d-456d-bf27-90063f9970e6-proxy-tls\") pod \"a8a717cc-199d-456d-bf27-90063f9970e6\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") "
Sep 17 18:04:22.904417 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.904183    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsb77\" (UniqueName: \"kubernetes.io/projected/a8a717cc-199d-456d-bf27-90063f9970e6-kube-api-access-nsb77\") pod \"a8a717cc-199d-456d-bf27-90063f9970e6\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") "
Sep 17 18:04:22.904417 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.904214    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-kube-rbac-proxy-sar-config\") pod \"a8a717cc-199d-456d-bf27-90063f9970e6\" (UID: \"a8a717cc-199d-456d-bf27-90063f9970e6\") "
Sep 17 18:04:22.904495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.904430    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a8a717cc-199d-456d-bf27-90063f9970e6" (UID: "a8a717cc-199d-456d-bf27-90063f9970e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:22.904625 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.904601    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8a717cc-199d-456d-bf27-90063f9970e6" (UID: "a8a717cc-199d-456d-bf27-90063f9970e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:22.906230 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.906206    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a717cc-199d-456d-bf27-90063f9970e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8a717cc-199d-456d-bf27-90063f9970e6" (UID: "a8a717cc-199d-456d-bf27-90063f9970e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:22.906337 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.906300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a717cc-199d-456d-bf27-90063f9970e6-kube-api-access-nsb77" (OuterVolumeSpecName: "kube-api-access-nsb77") pod "a8a717cc-199d-456d-bf27-90063f9970e6" (UID: "a8a717cc-199d-456d-bf27-90063f9970e6"). InnerVolumeSpecName "kube-api-access-nsb77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:22.926810 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:22.926789    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a717cc-199d-456d-bf27-90063f9970e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a8a717cc-199d-456d-bf27-90063f9970e6" (UID: "a8a717cc-199d-456d-bf27-90063f9970e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:23.004753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.004731    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:04:23.004753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.004751    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a8a717cc-199d-456d-bf27-90063f9970e6-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:04:23.004872 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.004760    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8a717cc-199d-456d-bf27-90063f9970e6-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:04:23.004872 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.004768    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsb77\" (UniqueName: \"kubernetes.io/projected/a8a717cc-199d-456d-bf27-90063f9970e6-kube-api-access-nsb77\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:04:23.004872 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.004778    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8a717cc-199d-456d-bf27-90063f9970e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:04:23.669531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.669444    2562 generic.go:356] "Generic (PLEG): container finished" podID="a8a717cc-199d-456d-bf27-90063f9970e6" containerID="ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea" exitCode=0
Sep 17 18:04:23.669531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.669480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerDied","Data":"ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea"}
Sep 17 18:04:23.669531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.669507    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"
Sep 17 18:04:23.669805 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.669534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr" event={"ID":"a8a717cc-199d-456d-bf27-90063f9970e6","Type":"ContainerDied","Data":"d485a0872375e79fc0948432128973c2a034555db3268c86aadc10fa7b7995f5"}
Sep 17 18:04:23.669805 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.669562    2562 scope.go:117] "RemoveContainer" containerID="7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8"
Sep 17 18:04:23.677187 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.677172    2562 scope.go:117] "RemoveContainer" containerID="ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea"
Sep 17 18:04:23.683859 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.683843    2562 scope.go:117] "RemoveContainer" containerID="d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19"
Sep 17 18:04:23.688486 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.688465    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"]
Sep 17 18:04:23.691178 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.691161    2562 scope.go:117] "RemoveContainer" containerID="7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8"
Sep 17 18:04:23.691452 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:04:23.691432    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8\": container with ID starting with 7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8 not found: ID does not exist" containerID="7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8"
Sep 17 18:04:23.691520 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.691460    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8"} err="failed to get container status \"7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8\": rpc error: code = NotFound desc = could not find container \"7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8\": container with ID starting with 7808bce4b65be44cecb61a623a7337f4ecb80bb7f0197e34bd0cf0c7629158e8 not found: ID does not exist"
Sep 17 18:04:23.691520 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.691476    2562 scope.go:117] "RemoveContainer" containerID="ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea"
Sep 17 18:04:23.691736 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:04:23.691722    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea\": container with ID starting with ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea not found: ID does not exist" containerID="ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea"
Sep 17 18:04:23.691790 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.691738    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea"} err="failed to get container status \"ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea\": rpc error: code = NotFound desc = could not find container \"ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea\": container with ID starting with ce367b6c6c81dc60e5377067ac0213e199537c9f6a2e512ec2f7fce8308dfbea not found: ID does not exist"
Sep 17 18:04:23.691790 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.691749    2562 scope.go:117] "RemoveContainer" containerID="d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19"
Sep 17 18:04:23.691902 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.691817    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6cdb8db44c-xcbpr"]
Sep 17 18:04:23.692014 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:04:23.691992    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19\": container with ID starting with d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19 not found: ID does not exist" containerID="d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19"
Sep 17 18:04:23.692063 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:23.692021    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19"} err="failed to get container status \"d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19\": rpc error: code = NotFound desc = could not find container \"d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19\": container with ID starting with d5144104762fb1747197c813cba521a29699579a65102cdc4bea7e309c70ed19 not found: ID does not exist"
Sep 17 18:04:25.276311 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:25.276278    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" path="/var/lib/kubelet/pods/a8a717cc-199d-456d-bf27-90063f9970e6/volumes"
Sep 17 18:04:54.305602 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:04:54.305559    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:05:05.312199 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:05.312098    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.28:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:05:09.833051 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:09.833020    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"]
Sep 17 18:05:09.833467 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:09.833274    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kserve-container" containerID="cri-o://aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774" gracePeriod=30
Sep 17 18:05:09.833467 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:09.833315    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kube-rbac-proxy" containerID="cri-o://90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26" gracePeriod=30
Sep 17 18:05:10.840020 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:10.839984    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a460157-321e-4514-a15c-a55e23ddb707" containerID="90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26" exitCode=2
Sep 17 18:05:10.840583 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:10.840056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerDied","Data":"90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26"}
Sep 17 18:05:11.267300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.267272    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:05:11.368861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.368774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-cabundle-cert\") pod \"4a460157-321e-4514-a15c-a55e23ddb707\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") "
Sep 17 18:05:11.368861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.368812    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a460157-321e-4514-a15c-a55e23ddb707-kserve-provision-location\") pod \"4a460157-321e-4514-a15c-a55e23ddb707\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") "
Sep 17 18:05:11.368861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.368834    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a460157-321e-4514-a15c-a55e23ddb707-proxy-tls\") pod \"4a460157-321e-4514-a15c-a55e23ddb707\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") "
Sep 17 18:05:11.368861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.368852    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-kube-rbac-proxy-sar-config\") pod \"4a460157-321e-4514-a15c-a55e23ddb707\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") "
Sep 17 18:05:11.369215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.368879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gnsn\" (UniqueName: \"kubernetes.io/projected/4a460157-321e-4514-a15c-a55e23ddb707-kube-api-access-5gnsn\") pod \"4a460157-321e-4514-a15c-a55e23ddb707\" (UID: \"4a460157-321e-4514-a15c-a55e23ddb707\") "
Sep 17 18:05:11.369269 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.369243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a460157-321e-4514-a15c-a55e23ddb707" (UID: "4a460157-321e-4514-a15c-a55e23ddb707"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:11.369269 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.369252    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a460157-321e-4514-a15c-a55e23ddb707" (UID: "4a460157-321e-4514-a15c-a55e23ddb707"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:11.370938 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.370898    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a460157-321e-4514-a15c-a55e23ddb707-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a460157-321e-4514-a15c-a55e23ddb707" (UID: "4a460157-321e-4514-a15c-a55e23ddb707"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:11.371056 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.371008    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a460157-321e-4514-a15c-a55e23ddb707-kube-api-access-5gnsn" (OuterVolumeSpecName: "kube-api-access-5gnsn") pod "4a460157-321e-4514-a15c-a55e23ddb707" (UID: "4a460157-321e-4514-a15c-a55e23ddb707"). InnerVolumeSpecName "kube-api-access-5gnsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:11.373097 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.373072    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a460157-321e-4514-a15c-a55e23ddb707-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a460157-321e-4514-a15c-a55e23ddb707" (UID: "4a460157-321e-4514-a15c-a55e23ddb707"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:11.469771 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.469743    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:05:11.469771 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.469767    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a460157-321e-4514-a15c-a55e23ddb707-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:05:11.469771 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.469778    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a460157-321e-4514-a15c-a55e23ddb707-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:05:11.469990 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.469787    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a460157-321e-4514-a15c-a55e23ddb707-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:05:11.469990 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.469795    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gnsn\" (UniqueName: \"kubernetes.io/projected/4a460157-321e-4514-a15c-a55e23ddb707-kube-api-access-5gnsn\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:05:11.846383 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.846343    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a460157-321e-4514-a15c-a55e23ddb707" containerID="aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774" exitCode=0
Sep 17 18:05:11.846757 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.846400    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"
Sep 17 18:05:11.846757 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.846421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerDied","Data":"aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774"}
Sep 17 18:05:11.846757 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.846469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49" event={"ID":"4a460157-321e-4514-a15c-a55e23ddb707","Type":"ContainerDied","Data":"41d7f6e38e786427880a5be2ff701fe9a52a29b9d00ce6c83c945b41356d10cb"}
Sep 17 18:05:11.846757 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.846498    2562 scope.go:117] "RemoveContainer" containerID="90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26"
Sep 17 18:05:11.855076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.855053    2562 scope.go:117] "RemoveContainer" containerID="aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774"
Sep 17 18:05:11.862088 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.862073    2562 scope.go:117] "RemoveContainer" containerID="ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d"
Sep 17 18:05:11.868531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.868509    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"]
Sep 17 18:05:11.869000 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.868984    2562 scope.go:117] "RemoveContainer" containerID="90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26"
Sep 17 18:05:11.869261 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:05:11.869243    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26\": container with ID starting with 90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26 not found: ID does not exist" containerID="90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26"
Sep 17 18:05:11.869333 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.869273    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26"} err="failed to get container status \"90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26\": rpc error: code = NotFound desc = could not find container \"90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26\": container with ID starting with 90dece8ce0c4880ef7215b11cfc692e1291fed2c218b49378a526a3c54fbaa26 not found: ID does not exist"
Sep 17 18:05:11.869333 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.869298    2562 scope.go:117] "RemoveContainer" containerID="aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774"
Sep 17 18:05:11.869554 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:05:11.869537    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774\": container with ID starting with aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774 not found: ID does not exist" containerID="aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774"
Sep 17 18:05:11.869620 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.869561    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774"} err="failed to get container status \"aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774\": rpc error: code = NotFound desc = could not find container \"aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774\": container with ID starting with aea1f408fc0a84d9821960d17758b0459e88c2aa7167e03930fca0d4c6a42774 not found: ID does not exist"
Sep 17 18:05:11.869620 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.869582    2562 scope.go:117] "RemoveContainer" containerID="ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d"
Sep 17 18:05:11.869845 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:05:11.869826    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d\": container with ID starting with ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d not found: ID does not exist" containerID="ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d"
Sep 17 18:05:11.869882 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.869851    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d"} err="failed to get container status \"ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d\": rpc error: code = NotFound desc = could not find container \"ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d\": container with ID starting with ef08df757801629accfaad5c7d650e2b05f0fb8513f7d98790294238411ba42d not found: ID does not exist"
Sep 17 18:05:11.873148 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:11.873127    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-bkw49"]
Sep 17 18:05:13.276154 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:13.276115    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a460157-321e-4514-a15c-a55e23ddb707" path="/var/lib/kubelet/pods/4a460157-321e-4514-a15c-a55e23ddb707/volumes"
Sep 17 18:05:32.257282 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257239    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"]
Sep 17 18:05:32.257732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257654    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kube-rbac-proxy"
Sep 17 18:05:32.257732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257671    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kube-rbac-proxy"
Sep 17 18:05:32.257732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257685    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kube-rbac-proxy"
Sep 17 18:05:32.257732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257694    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kube-rbac-proxy"
Sep 17 18:05:32.257732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257711    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="storage-initializer"
Sep 17 18:05:32.257732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257720    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="storage-initializer"
Sep 17 18:05:32.257732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257734    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kserve-container"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257755    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kserve-container"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257766    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kserve-container"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257776    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kserve-container"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257787    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="storage-initializer"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257796    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="storage-initializer"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257855    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kserve-container"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257866    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a460157-321e-4514-a15c-a55e23ddb707" containerName="kube-rbac-proxy"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257877    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kserve-container"
Sep 17 18:05:32.258093 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.257885    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8a717cc-199d-456d-bf27-90063f9970e6" containerName="kube-rbac-proxy"
Sep 17 18:05:32.262727 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.262706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.265733 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.265699    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:05:32.266397 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.266377    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:05:32.266508 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.266395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 18:05:32.266508 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.266378    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:05:32.266508 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.266378    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:05:32.266684 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.266532    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:05:32.266684 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.266530    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:05:32.275608 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.275583    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"]
Sep 17 18:05:32.317759 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.317724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.317759 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.317764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.318011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.317785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.318011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.317857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.318011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.317936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc9mc\" (UniqueName: \"kubernetes.io/projected/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-api-access-sc9mc\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419025 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.418968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419025 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.419034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.419061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.419083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.419106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc9mc\" (UniqueName: \"kubernetes.io/projected/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-api-access-sc9mc\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419572 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.419544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419682 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.419664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.419786 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.419767    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.421614 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.421594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-proxy-tls\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.427639 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.427619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc9mc\" (UniqueName: \"kubernetes.io/projected/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-api-access-sc9mc\") pod \"isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.572884 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.572774    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:32.846317 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.846286    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"]
Sep 17 18:05:32.848664 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:05:32.848637    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34de20b8_8339_476f_9a0a_49dfc7f3b8f7.slice/crio-3c19b6cd052918c83e383ba543d93df0ec520d9ea65c27354c1d0576c4636b1a WatchSource:0}: Error finding container 3c19b6cd052918c83e383ba543d93df0ec520d9ea65c27354c1d0576c4636b1a: Status 404 returned error can't find the container with id 3c19b6cd052918c83e383ba543d93df0ec520d9ea65c27354c1d0576c4636b1a
Sep 17 18:05:32.920559 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.920514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerStarted","Data":"2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49"}
Sep 17 18:05:32.920710 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:32.920599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerStarted","Data":"3c19b6cd052918c83e383ba543d93df0ec520d9ea65c27354c1d0576c4636b1a"}
Sep 17 18:05:36.936653 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:36.936572    2562 generic.go:356] "Generic (PLEG): container finished" podID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerID="2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49" exitCode=0
Sep 17 18:05:36.936653 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:36.936611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerDied","Data":"2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49"}
Sep 17 18:05:37.942629 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:37.942593    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerStarted","Data":"6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c"}
Sep 17 18:05:37.943010 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:37.942659    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerStarted","Data":"ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a"}
Sep 17 18:05:37.943010 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:37.942854    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:37.965367 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:37.965327    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podStartSLOduration=2.184680224 podStartE2EDuration="5.965313388s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780633164s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:37.964475039 +0000 UTC m=+1829.151910889" watchObservedRunningTime="2026-09-17 18:05:37.965313388 +0000 UTC m=+1829.152749236"
Sep 17 18:05:38.946266 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:38.946234    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:38.947463 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:38.947434    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:05:39.949568 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:39.949524    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:05:44.954621 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:44.954589    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:05:44.955289 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:44.955260    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:05:54.955694 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:05:54.955649    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:06:03.017203 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.017165    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"]
Sep 17 18:06:03.021220 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.021192    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.024209 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.024180    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 18:06:03.024374 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.024207    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:06:03.036785 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.036758    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"]
Sep 17 18:06:03.137361 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.137322    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.137361 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.137370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lqh\" (UniqueName: \"kubernetes.io/projected/54112abb-9882-44e4-9632-71bd18385161-kube-api-access-t4lqh\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.137577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.137394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.137577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.137490    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54112abb-9882-44e4-9632-71bd18385161-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.137577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.137521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54112abb-9882-44e4-9632-71bd18385161-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.238333 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.238286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.238333 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.238339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lqh\" (UniqueName: \"kubernetes.io/projected/54112abb-9882-44e4-9632-71bd18385161-kube-api-access-t4lqh\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.238608 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.238362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.238608 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.238399    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54112abb-9882-44e4-9632-71bd18385161-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.238608 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.238417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54112abb-9882-44e4-9632-71bd18385161-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.238849 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.238827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54112abb-9882-44e4-9632-71bd18385161-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.239081 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.239056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.239201 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.239130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.241004 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.240983    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54112abb-9882-44e4-9632-71bd18385161-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.246987 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.246969    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lqh\" (UniqueName: \"kubernetes.io/projected/54112abb-9882-44e4-9632-71bd18385161-kube-api-access-t4lqh\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.331853 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.331779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:03.605285 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:03.605212    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"]
Sep 17 18:06:03.607755 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:06:03.607727    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54112abb_9882_44e4_9632_71bd18385161.slice/crio-07b80f9b142ecca99c06a53ae6cf3fa8002e16db18619c434e6718d23980c197 WatchSource:0}: Error finding container 07b80f9b142ecca99c06a53ae6cf3fa8002e16db18619c434e6718d23980c197: Status 404 returned error can't find the container with id 07b80f9b142ecca99c06a53ae6cf3fa8002e16db18619c434e6718d23980c197
Sep 17 18:06:04.037708 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:04.037669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerStarted","Data":"08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a"}
Sep 17 18:06:04.038184 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:04.037744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerStarted","Data":"07b80f9b142ecca99c06a53ae6cf3fa8002e16db18619c434e6718d23980c197"}
Sep 17 18:06:04.955745 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:04.955703    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:06:08.055696 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:08.055660    2562 generic.go:356] "Generic (PLEG): container finished" podID="54112abb-9882-44e4-9632-71bd18385161" containerID="08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a" exitCode=0
Sep 17 18:06:08.056183 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:08.056154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerDied","Data":"08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a"}
Sep 17 18:06:09.063592 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:09.063560    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerStarted","Data":"75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952"}
Sep 17 18:06:09.063991 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:09.063622    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerStarted","Data":"c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70"}
Sep 17 18:06:09.063991 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:09.063825    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:09.086404 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:09.086343    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" podStartSLOduration=2.714174159 podStartE2EDuration="7.086321411s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.372147252s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:09.084458578 +0000 UTC m=+1860.271894441" watchObservedRunningTime="2026-09-17 18:06:09.086321411 +0000 UTC m=+1860.273757264"
Sep 17 18:06:10.067359 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:10.067324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:10.068465 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:10.068441    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:06:11.071092 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:11.071046    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:06:14.956009 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:14.955967    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:06:16.076052 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:16.076015    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:16.076608 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:16.076585    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 18:06:24.955494 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:24.955455    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:06:26.078039 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:26.078010    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:34.955705 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:34.955623    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:06:38.311229 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:38.311193    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"]
Sep 17 18:06:38.311603 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:38.311491    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kserve-container" containerID="cri-o://c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70" gracePeriod=30
Sep 17 18:06:38.311603 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:38.311564    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kube-rbac-proxy" containerID="cri-o://75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952" gracePeriod=30
Sep 17 18:06:39.184636 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.184599    2562 generic.go:356] "Generic (PLEG): container finished" podID="54112abb-9882-44e4-9632-71bd18385161" containerID="75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952" exitCode=2
Sep 17 18:06:39.184825 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.184657    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerDied","Data":"75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952"}
Sep 17 18:06:39.751085 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.751058    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:39.929875 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.929775    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54112abb-9882-44e4-9632-71bd18385161-kserve-provision-location\") pod \"54112abb-9882-44e4-9632-71bd18385161\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") "
Sep 17 18:06:39.929875 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.929842    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-kube-rbac-proxy-sar-config\") pod \"54112abb-9882-44e4-9632-71bd18385161\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") "
Sep 17 18:06:39.929875 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.929877    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4lqh\" (UniqueName: \"kubernetes.io/projected/54112abb-9882-44e4-9632-71bd18385161-kube-api-access-t4lqh\") pod \"54112abb-9882-44e4-9632-71bd18385161\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") "
Sep 17 18:06:39.930170 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.929950    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54112abb-9882-44e4-9632-71bd18385161-proxy-tls\") pod \"54112abb-9882-44e4-9632-71bd18385161\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") "
Sep 17 18:06:39.930170 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.929982    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-cabundle-cert\") pod \"54112abb-9882-44e4-9632-71bd18385161\" (UID: \"54112abb-9882-44e4-9632-71bd18385161\") "
Sep 17 18:06:39.930358 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.930253    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "54112abb-9882-44e4-9632-71bd18385161" (UID: "54112abb-9882-44e4-9632-71bd18385161"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:39.930486 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.930414    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "54112abb-9882-44e4-9632-71bd18385161" (UID: "54112abb-9882-44e4-9632-71bd18385161"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:39.931019 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.931000    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54112abb-9882-44e4-9632-71bd18385161-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "54112abb-9882-44e4-9632-71bd18385161" (UID: "54112abb-9882-44e4-9632-71bd18385161"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:39.932113 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.932087    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54112abb-9882-44e4-9632-71bd18385161-kube-api-access-t4lqh" (OuterVolumeSpecName: "kube-api-access-t4lqh") pod "54112abb-9882-44e4-9632-71bd18385161" (UID: "54112abb-9882-44e4-9632-71bd18385161"). InnerVolumeSpecName "kube-api-access-t4lqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:39.932230 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:39.932207    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54112abb-9882-44e4-9632-71bd18385161-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54112abb-9882-44e4-9632-71bd18385161" (UID: "54112abb-9882-44e4-9632-71bd18385161"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:40.030903 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.030866    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/54112abb-9882-44e4-9632-71bd18385161-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:06:40.030903 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.030897    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:06:40.030903 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.030907    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4lqh\" (UniqueName: \"kubernetes.io/projected/54112abb-9882-44e4-9632-71bd18385161-kube-api-access-t4lqh\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:06:40.031169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.030935    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54112abb-9882-44e4-9632-71bd18385161-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:06:40.031169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.030945    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/54112abb-9882-44e4-9632-71bd18385161-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:06:40.194016 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.193928    2562 generic.go:356] "Generic (PLEG): container finished" podID="54112abb-9882-44e4-9632-71bd18385161" containerID="c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70" exitCode=0
Sep 17 18:06:40.194016 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.193983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerDied","Data":"c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70"}
Sep 17 18:06:40.194016 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.194000    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"
Sep 17 18:06:40.194246 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.194071    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc" event={"ID":"54112abb-9882-44e4-9632-71bd18385161","Type":"ContainerDied","Data":"07b80f9b142ecca99c06a53ae6cf3fa8002e16db18619c434e6718d23980c197"}
Sep 17 18:06:40.194246 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.194109    2562 scope.go:117] "RemoveContainer" containerID="75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952"
Sep 17 18:06:40.202637 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.202620    2562 scope.go:117] "RemoveContainer" containerID="c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70"
Sep 17 18:06:40.209640 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.209621    2562 scope.go:117] "RemoveContainer" containerID="08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a"
Sep 17 18:06:40.216016 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.215997    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"]
Sep 17 18:06:40.216679 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.216663    2562 scope.go:117] "RemoveContainer" containerID="75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952"
Sep 17 18:06:40.217011 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:06:40.216901    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952\": container with ID starting with 75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952 not found: ID does not exist" containerID="75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952"
Sep 17 18:06:40.217011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.216943    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952"} err="failed to get container status \"75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952\": rpc error: code = NotFound desc = could not find container \"75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952\": container with ID starting with 75c94ca84849010a869db07a3e7470d264404eae9e8b6b7f594c49fb35e9a952 not found: ID does not exist"
Sep 17 18:06:40.217011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.216964    2562 scope.go:117] "RemoveContainer" containerID="c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70"
Sep 17 18:06:40.217227 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:06:40.217209    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70\": container with ID starting with c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70 not found: ID does not exist" containerID="c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70"
Sep 17 18:06:40.217280 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.217232    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70"} err="failed to get container status \"c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70\": rpc error: code = NotFound desc = could not find container \"c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70\": container with ID starting with c3592056f44e986c1893e2def1ad3800667725a0064e9cce1230ae38a5d2eb70 not found: ID does not exist"
Sep 17 18:06:40.217280 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.217249    2562 scope.go:117] "RemoveContainer" containerID="08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a"
Sep 17 18:06:40.217481 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:06:40.217466    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a\": container with ID starting with 08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a not found: ID does not exist" containerID="08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a"
Sep 17 18:06:40.217529 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.217484    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a"} err="failed to get container status \"08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a\": rpc error: code = NotFound desc = could not find container \"08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a\": container with ID starting with 08d924d9fc6e57f4c67477b16e8be2578423f1d6c82aa10c51ec3ef2a78efe1a not found: ID does not exist"
Sep 17 18:06:40.219978 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:40.219950    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jfbjc"]
Sep 17 18:06:41.275529 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:41.275497    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54112abb-9882-44e4-9632-71bd18385161" path="/var/lib/kubelet/pods/54112abb-9882-44e4-9632-71bd18385161/volumes"
Sep 17 18:06:43.497219 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497186    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"]
Sep 17 18:06:43.497568 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497486    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kube-rbac-proxy"
Sep 17 18:06:43.497568 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497503    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kube-rbac-proxy"
Sep 17 18:06:43.497568 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497513    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="storage-initializer"
Sep 17 18:06:43.497568 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497519    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="storage-initializer"
Sep 17 18:06:43.497568 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497528    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kserve-container"
Sep 17 18:06:43.497568 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497534    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kserve-container"
Sep 17 18:06:43.497758 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497586    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kserve-container"
Sep 17 18:06:43.497758 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.497595    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="54112abb-9882-44e4-9632-71bd18385161" containerName="kube-rbac-proxy"
Sep 17 18:06:43.502324 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.502302    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.504497 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.504477    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:06:43.504602 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.504484    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 18:06:43.513777 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.513755    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"]
Sep 17 18:06:43.532446 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.532394    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"]
Sep 17 18:06:43.535516 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.535499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.537540 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.537518    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:06:43.546358 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.546336    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"]
Sep 17 18:06:43.556255 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b9983a-e665-4916-baab-5aacadbab524-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.556361 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.556361 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.556361 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.556482 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.556482 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbjmw\" (UniqueName: \"kubernetes.io/projected/71b9983a-e665-4916-baab-5aacadbab524-kube-api-access-kbjmw\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.556482 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbch\" (UniqueName: \"kubernetes.io/projected/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-api-access-xbbch\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.556482 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9983a-e665-4916-baab-5aacadbab524-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.556623 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.556623 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.556532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.657586 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbjmw\" (UniqueName: \"kubernetes.io/projected/71b9983a-e665-4916-baab-5aacadbab524-kube-api-access-kbjmw\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.657586 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbch\" (UniqueName: \"kubernetes.io/projected/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-api-access-xbbch\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.657794 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9983a-e665-4916-baab-5aacadbab524-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.657794 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.657794 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.657794 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657694    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b9983a-e665-4916-baab-5aacadbab524-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.658021 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.658021 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.658021 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:06:43.657893    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:06:43.658021 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657901    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.658021 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.657949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.658021 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:06:43.657994    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls podName:f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0 nodeName:}" failed. No retries permitted until 2026-09-17 18:06:44.157971642 +0000 UTC m=+1895.345407482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls") pod "isvc-canary-create-predictor-7d998654b-fqs9q" (UID: "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 17 18:06:43.658299 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.658242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9983a-e665-4916-baab-5aacadbab524-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.658542 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.658516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.658628 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.658595    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-cabundle-cert\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.658672 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.658643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.658672 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.658647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.658746 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.658723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.660257 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.660236    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b9983a-e665-4916-baab-5aacadbab524-proxy-tls\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.673977 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.673957    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbch\" (UniqueName: \"kubernetes.io/projected/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-api-access-xbbch\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:43.675067 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.675045    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbjmw\" (UniqueName: \"kubernetes.io/projected/71b9983a-e665-4916-baab-5aacadbab524-kube-api-access-kbjmw\") pod \"isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:43.812162 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:43.812085    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:44.088055 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.087975    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"]
Sep 17 18:06:44.090899 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:06:44.090874    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b9983a_e665_4916_baab_5aacadbab524.slice/crio-5fd28e2f2af85b281721b93f467399d14521827f68a0c0bf0d12ad1e11cf7c38 WatchSource:0}: Error finding container 5fd28e2f2af85b281721b93f467399d14521827f68a0c0bf0d12ad1e11cf7c38: Status 404 returned error can't find the container with id 5fd28e2f2af85b281721b93f467399d14521827f68a0c0bf0d12ad1e11cf7c38
Sep 17 18:06:44.163372 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.163346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:44.165396 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.165372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls\") pod \"isvc-canary-create-predictor-7d998654b-fqs9q\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:44.215437 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.215404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerStarted","Data":"7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48"}
Sep 17 18:06:44.215584 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.215488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerStarted","Data":"5fd28e2f2af85b281721b93f467399d14521827f68a0c0bf0d12ad1e11cf7c38"}
Sep 17 18:06:44.444948 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.444854    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:44.722352 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.722324    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"]
Sep 17 18:06:44.725028 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:06:44.724992    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bb6598_c5cc_4ce3_ac27_54f2f5c4dbc0.slice/crio-5359bfc1be1a660a92db6c3cf938fd3f9cceb21d0f8002ed572436a34604d4db WatchSource:0}: Error finding container 5359bfc1be1a660a92db6c3cf938fd3f9cceb21d0f8002ed572436a34604d4db: Status 404 returned error can't find the container with id 5359bfc1be1a660a92db6c3cf938fd3f9cceb21d0f8002ed572436a34604d4db
Sep 17 18:06:44.955734 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:44.955688    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:06:45.221582 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:45.221537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerStarted","Data":"93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd"}
Sep 17 18:06:45.221778 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:45.221616    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerStarted","Data":"5359bfc1be1a660a92db6c3cf938fd3f9cceb21d0f8002ed572436a34604d4db"}
Sep 17 18:06:48.235721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:48.235691    2562 generic.go:356] "Generic (PLEG): container finished" podID="71b9983a-e665-4916-baab-5aacadbab524" containerID="7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48" exitCode=0
Sep 17 18:06:48.236021 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:48.235756    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerDied","Data":"7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48"}
Sep 17 18:06:49.241290 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.241259    2562 generic.go:356] "Generic (PLEG): container finished" podID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerID="93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd" exitCode=0
Sep 17 18:06:49.241690 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.241338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerDied","Data":"93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd"}
Sep 17 18:06:49.243477 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.243458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerStarted","Data":"f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1"}
Sep 17 18:06:49.243650 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.243501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerStarted","Data":"9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db"}
Sep 17 18:06:49.243788 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.243773    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:49.243878 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.243826    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:49.244842 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.244822    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:06:49.289445 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:49.289404    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podStartSLOduration=2.220446039 podStartE2EDuration="6.289390489s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.06894445s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:49.28653558 +0000 UTC m=+1900.473971428" watchObservedRunningTime="2026-09-17 18:06:49.289390489 +0000 UTC m=+1900.476826337"
Sep 17 18:06:50.249682 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:50.249639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerStarted","Data":"93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da"}
Sep 17 18:06:50.250118 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:50.249756    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerStarted","Data":"3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1"}
Sep 17 18:06:50.250118 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:50.250027    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:06:50.250118 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:50.250054    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:50.269670 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:50.269624    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podStartSLOduration=3.4773989370000002 podStartE2EDuration="7.269611166s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.792212229s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:50.268905021 +0000 UTC m=+1901.456340883" watchObservedRunningTime="2026-09-17 18:06:50.269611166 +0000 UTC m=+1901.457047015"
Sep 17 18:06:51.254335 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:51.254299    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:51.255539 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:51.255508    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:06:52.259224 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:52.259184    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:06:54.955892 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:54.955862    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:06:55.256066 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:55.256039    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:06:55.256602 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:55.256563    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:06:57.265076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:57.265050    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:06:57.265561 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:06:57.265534    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:07:02.365605 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:02.365569    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"]
Sep 17 18:07:02.366032 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:02.365817    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" containerID="cri-o://ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a" gracePeriod=30
Sep 17 18:07:02.366032 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:02.365852    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kube-rbac-proxy" containerID="cri-o://6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c" gracePeriod=30
Sep 17 18:07:03.299974 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:03.299940    2562 generic.go:356] "Generic (PLEG): container finished" podID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerID="6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c" exitCode=2
Sep 17 18:07:03.300143 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:03.300007    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerDied","Data":"6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c"}
Sep 17 18:07:04.950749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:04.950701    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 18:07:04.955223 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:04.955180    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 18:07:05.257272 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:05.257231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:07:07.005653 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.005629    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:07:07.038164 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038126    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kserve-provision-location\") pod \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") "
Sep 17 18:07:07.038300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc9mc\" (UniqueName: \"kubernetes.io/projected/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-api-access-sc9mc\") pod \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") "
Sep 17 18:07:07.038300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038225    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-cabundle-cert\") pod \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") "
Sep 17 18:07:07.038300 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038270    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-proxy-tls\") pod \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") "
Sep 17 18:07:07.038464 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038332    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-rbac-proxy-sar-config\") pod \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\" (UID: \"34de20b8-8339-476f-9a0a-49dfc7f3b8f7\") "
Sep 17 18:07:07.038464 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038411    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34de20b8-8339-476f-9a0a-49dfc7f3b8f7" (UID: "34de20b8-8339-476f-9a0a-49dfc7f3b8f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:07.038581 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038558    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:07:07.038677 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038656    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34de20b8-8339-476f-9a0a-49dfc7f3b8f7" (UID: "34de20b8-8339-476f-9a0a-49dfc7f3b8f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:07.038727 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.038691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34de20b8-8339-476f-9a0a-49dfc7f3b8f7" (UID: "34de20b8-8339-476f-9a0a-49dfc7f3b8f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:07.040284 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.040257    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-api-access-sc9mc" (OuterVolumeSpecName: "kube-api-access-sc9mc") pod "34de20b8-8339-476f-9a0a-49dfc7f3b8f7" (UID: "34de20b8-8339-476f-9a0a-49dfc7f3b8f7"). InnerVolumeSpecName "kube-api-access-sc9mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:07.040408 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.040330    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34de20b8-8339-476f-9a0a-49dfc7f3b8f7" (UID: "34de20b8-8339-476f-9a0a-49dfc7f3b8f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:07.139076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.138981    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:07:07.139076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.139015    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc9mc\" (UniqueName: \"kubernetes.io/projected/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-kube-api-access-sc9mc\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:07:07.139076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.139028    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:07:07.139076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.139040    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34de20b8-8339-476f-9a0a-49dfc7f3b8f7-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:07:07.265923 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.265884    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:07:07.315826 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.315796    2562 generic.go:356] "Generic (PLEG): container finished" podID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerID="ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a" exitCode=0
Sep 17 18:07:07.315977 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.315827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerDied","Data":"ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a"}
Sep 17 18:07:07.315977 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.315864    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"
Sep 17 18:07:07.315977 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.315901    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2" event={"ID":"34de20b8-8339-476f-9a0a-49dfc7f3b8f7","Type":"ContainerDied","Data":"3c19b6cd052918c83e383ba543d93df0ec520d9ea65c27354c1d0576c4636b1a"}
Sep 17 18:07:07.315977 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.315945    2562 scope.go:117] "RemoveContainer" containerID="6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c"
Sep 17 18:07:07.323641 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.323622    2562 scope.go:117] "RemoveContainer" containerID="ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a"
Sep 17 18:07:07.330763 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.330742    2562 scope.go:117] "RemoveContainer" containerID="2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49"
Sep 17 18:07:07.337722 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.337702    2562 scope.go:117] "RemoveContainer" containerID="6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c"
Sep 17 18:07:07.337985 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:07:07.337962    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c\": container with ID starting with 6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c not found: ID does not exist" containerID="6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c"
Sep 17 18:07:07.338053 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.337994    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c"} err="failed to get container status \"6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c\": rpc error: code = NotFound desc = could not find container \"6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c\": container with ID starting with 6add0a76a1d1f4200931f6cabc6af06c29d1b69666174c026bbbecfc8351237c not found: ID does not exist"
Sep 17 18:07:07.338053 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.338011    2562 scope.go:117] "RemoveContainer" containerID="ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a"
Sep 17 18:07:07.338262 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:07:07.338246    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a\": container with ID starting with ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a not found: ID does not exist" containerID="ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a"
Sep 17 18:07:07.338303 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.338269    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a"} err="failed to get container status \"ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a\": rpc error: code = NotFound desc = could not find container \"ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a\": container with ID starting with ed56513dcbfc0e3b02c8c599a0cafb4a7598415cc866b89f2ce981f21f741b8a not found: ID does not exist"
Sep 17 18:07:07.338303 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.338284    2562 scope.go:117] "RemoveContainer" containerID="2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49"
Sep 17 18:07:07.338528 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:07:07.338510    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49\": container with ID starting with 2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49 not found: ID does not exist" containerID="2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49"
Sep 17 18:07:07.338582 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.338531    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49"} err="failed to get container status \"2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49\": rpc error: code = NotFound desc = could not find container \"2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49\": container with ID starting with 2e56b8560dba5b8ea1b793a65f4974be16a11a866da95f72aa62729736885f49 not found: ID does not exist"
Sep 17 18:07:07.345592 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.345568    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"]
Sep 17 18:07:07.359546 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.359524    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-65cb4cbbf5-wqjz2"]
Sep 17 18:07:07.615714 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.615681    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"]
Sep 17 18:07:07.615993 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.615980    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kube-rbac-proxy"
Sep 17 18:07:07.616045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.615995    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kube-rbac-proxy"
Sep 17 18:07:07.616045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.616011    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="storage-initializer"
Sep 17 18:07:07.616045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.616017    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="storage-initializer"
Sep 17 18:07:07.616045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.616024    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container"
Sep 17 18:07:07.616045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.616031    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container"
Sep 17 18:07:07.616201 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.616073    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kserve-container"
Sep 17 18:07:07.616201 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.616082    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" containerName="kube-rbac-proxy"
Sep 17 18:07:07.620532 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.620511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.622975 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.622947    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 17 18:07:07.623087 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.622962    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 17 18:07:07.634610 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.634585    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"]
Sep 17 18:07:07.743943 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.743891    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljp4\" (UniqueName: \"kubernetes.io/projected/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-api-access-vljp4\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.744108 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.743973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.744108 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.744004    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173303a9-982c-4524-93ef-b8d47bfa27d4-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.744217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.744114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.744217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.744148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173303a9-982c-4524-93ef-b8d47bfa27d4-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.845205 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.845167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.845205 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.845206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173303a9-982c-4524-93ef-b8d47bfa27d4-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.845439 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.845230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljp4\" (UniqueName: \"kubernetes.io/projected/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-api-access-vljp4\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.845439 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.845252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.845439 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.845366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173303a9-982c-4524-93ef-b8d47bfa27d4-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.850652 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.846235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173303a9-982c-4524-93ef-b8d47bfa27d4-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.850652 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.846351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.850652 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.846378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.850652 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.848857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173303a9-982c-4524-93ef-b8d47bfa27d4-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.859984 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.859950    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljp4\" (UniqueName: \"kubernetes.io/projected/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-api-access-vljp4\") pod \"isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:07.929974 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:07.929853    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:08.207500 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:08.207422    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"]
Sep 17 18:07:08.210529 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:07:08.210494    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173303a9_982c_4524_93ef_b8d47bfa27d4.slice/crio-4fb45e0947b65170f0a20aafc14888deb919f21d97d162e0f1e89bd4d281b70d WatchSource:0}: Error finding container 4fb45e0947b65170f0a20aafc14888deb919f21d97d162e0f1e89bd4d281b70d: Status 404 returned error can't find the container with id 4fb45e0947b65170f0a20aafc14888deb919f21d97d162e0f1e89bd4d281b70d
Sep 17 18:07:08.321649 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:08.321601    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerStarted","Data":"84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea"}
Sep 17 18:07:08.321851 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:08.321688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerStarted","Data":"4fb45e0947b65170f0a20aafc14888deb919f21d97d162e0f1e89bd4d281b70d"}
Sep 17 18:07:09.277371 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:09.277336    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34de20b8-8339-476f-9a0a-49dfc7f3b8f7" path="/var/lib/kubelet/pods/34de20b8-8339-476f-9a0a-49dfc7f3b8f7/volumes"
Sep 17 18:07:12.339601 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:12.339506    2562 generic.go:356] "Generic (PLEG): container finished" podID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerID="84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea" exitCode=0
Sep 17 18:07:12.340028 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:12.339590    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerDied","Data":"84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea"}
Sep 17 18:07:13.345875 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:13.345840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerStarted","Data":"f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151"}
Sep 17 18:07:13.346267 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:13.345907    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerStarted","Data":"10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83"}
Sep 17 18:07:13.346267 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:13.346106    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:13.373276 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:13.373227    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podStartSLOduration=2.622981557 podStartE2EDuration="6.373210112s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.750228555s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:13.371215143 +0000 UTC m=+1924.558650991" watchObservedRunningTime="2026-09-17 18:07:13.373210112 +0000 UTC m=+1924.560645962"
Sep 17 18:07:14.349552 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:14.349516    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:14.351017 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:14.350981    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:07:15.256963 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:15.256904    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:07:15.354846 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:15.354794    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:07:17.266112 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:17.266071    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:07:20.359971 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:20.359941    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:07:20.360486 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:20.360447    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:07:25.257395 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:25.257350    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:07:27.266001 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:27.265962    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:07:30.360856 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:30.360820    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:07:35.257387 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:35.257348    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:07:37.266139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:37.266099    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:07:40.361230 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:40.361192    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:07:45.256687 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:45.256640    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:07:47.266409 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:47.266370    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:07:50.360826 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:50.360784    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:07:55.257312 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:55.257271    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:07:57.266252 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:07:57.266205    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Sep 17 18:08:00.361202 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:00.361111    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:08:05.257121 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:05.257083    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:08:05.393241 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:05.393207    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"]
Sep 17 18:08:05.393466 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:05.393442    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container" containerID="cri-o://3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1" gracePeriod=30
Sep 17 18:08:05.393592 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:05.393558    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kube-rbac-proxy" containerID="cri-o://93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da" gracePeriod=30
Sep 17 18:08:05.394341 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:05.394078    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:08:05.539459 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:05.539427    2562 generic.go:356] "Generic (PLEG): container finished" podID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerID="93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da" exitCode=2
Sep 17 18:08:05.539618 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:05.539480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerDied","Data":"93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da"}
Sep 17 18:08:07.259749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:07.259706    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Sep 17 18:08:10.335841 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.335816    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:08:10.360503 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.360463    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:08:10.421350 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421256    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-cabundle-cert\") pod \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") "
Sep 17 18:08:10.421350 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-rbac-proxy-sar-config\") pod \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") "
Sep 17 18:08:10.421575 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421351    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbbch\" (UniqueName: \"kubernetes.io/projected/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-api-access-xbbch\") pod \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") "
Sep 17 18:08:10.421575 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls\") pod \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") "
Sep 17 18:08:10.421575 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421464    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kserve-provision-location\") pod \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\" (UID: \"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0\") "
Sep 17 18:08:10.421705 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421648    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" (UID: "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:10.421793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421689    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" (UID: "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:10.421892 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.421784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" (UID: "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:10.423456 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.423434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" (UID: "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:10.423516 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.423499    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-api-access-xbbch" (OuterVolumeSpecName: "kube-api-access-xbbch") pod "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" (UID: "f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0"). InnerVolumeSpecName "kube-api-access-xbbch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:10.522720 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.522672    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:10.522720 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.522713    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:10.522720 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.522723    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:10.522720 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.522732    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbbch\" (UniqueName: \"kubernetes.io/projected/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-kube-api-access-xbbch\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:10.522720 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.522741    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:10.564175 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.564144    2562 generic.go:356] "Generic (PLEG): container finished" podID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerID="3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1" exitCode=0
Sep 17 18:08:10.564326 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.564182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerDied","Data":"3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1"}
Sep 17 18:08:10.564326 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.564200    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"
Sep 17 18:08:10.564326 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.564254    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q" event={"ID":"f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0","Type":"ContainerDied","Data":"5359bfc1be1a660a92db6c3cf938fd3f9cceb21d0f8002ed572436a34604d4db"}
Sep 17 18:08:10.564326 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.564280    2562 scope.go:117] "RemoveContainer" containerID="93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da"
Sep 17 18:08:10.572198 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.572178    2562 scope.go:117] "RemoveContainer" containerID="3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1"
Sep 17 18:08:10.579140 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.579121    2562 scope.go:117] "RemoveContainer" containerID="93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd"
Sep 17 18:08:10.585081 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.585035    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"]
Sep 17 18:08:10.586987 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.586972    2562 scope.go:117] "RemoveContainer" containerID="93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da"
Sep 17 18:08:10.587291 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:10.587271    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da\": container with ID starting with 93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da not found: ID does not exist" containerID="93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da"
Sep 17 18:08:10.587358 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.587296    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da"} err="failed to get container status \"93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da\": rpc error: code = NotFound desc = could not find container \"93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da\": container with ID starting with 93b6846753375fce83fdbb62f7469c864ce62bdf0a64d99fa3f5d61a411383da not found: ID does not exist"
Sep 17 18:08:10.587358 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.587313    2562 scope.go:117] "RemoveContainer" containerID="3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1"
Sep 17 18:08:10.587642 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:10.587619    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1\": container with ID starting with 3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1 not found: ID does not exist" containerID="3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1"
Sep 17 18:08:10.587755 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.587651    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1"} err="failed to get container status \"3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1\": rpc error: code = NotFound desc = could not find container \"3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1\": container with ID starting with 3f9ad853e5828693578f0f4aa52b75a602f7bdc5bab32232e1f6881aebd9d5e1 not found: ID does not exist"
Sep 17 18:08:10.587755 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.587670    2562 scope.go:117] "RemoveContainer" containerID="93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd"
Sep 17 18:08:10.588412 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:10.588371    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd\": container with ID starting with 93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd not found: ID does not exist" containerID="93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd"
Sep 17 18:08:10.588501 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.588420    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd"} err="failed to get container status \"93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd\": rpc error: code = NotFound desc = could not find container \"93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd\": container with ID starting with 93f6e80182b82d94aa81dc0b7276c5879e418648ee1783d752faa31df1ddbefd not found: ID does not exist"
Sep 17 18:08:10.589250 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:10.589234    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d998654b-fqs9q"]
Sep 17 18:08:11.276551 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:11.276521    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" path="/var/lib/kubelet/pods/f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0/volumes"
Sep 17 18:08:20.360819 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:20.360781    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:08:23.583370 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:23.583335    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"]
Sep 17 18:08:23.583749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:23.583577    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" containerID="cri-o://9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db" gracePeriod=30
Sep 17 18:08:23.583749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:23.583648    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kube-rbac-proxy" containerID="cri-o://f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1" gracePeriod=30
Sep 17 18:08:24.617718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:24.617683    2562 generic.go:356] "Generic (PLEG): container finished" podID="71b9983a-e665-4916-baab-5aacadbab524" containerID="f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1" exitCode=2
Sep 17 18:08:24.618120 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:24.617750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerDied","Data":"f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1"}
Sep 17 18:08:25.251214 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:25.251161    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 18:08:25.256807 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:25.256774    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 18:08:28.223426 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.223401    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:08:28.360033 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.359947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b9983a-e665-4916-baab-5aacadbab524-proxy-tls\") pod \"71b9983a-e665-4916-baab-5aacadbab524\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") "
Sep 17 18:08:28.360033 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.359989    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbjmw\" (UniqueName: \"kubernetes.io/projected/71b9983a-e665-4916-baab-5aacadbab524-kube-api-access-kbjmw\") pod \"71b9983a-e665-4916-baab-5aacadbab524\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") "
Sep 17 18:08:28.360033 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.360019    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-cabundle-cert\") pod \"71b9983a-e665-4916-baab-5aacadbab524\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") "
Sep 17 18:08:28.360307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.360051    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-kube-rbac-proxy-sar-config\") pod \"71b9983a-e665-4916-baab-5aacadbab524\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") "
Sep 17 18:08:28.360307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.360072    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9983a-e665-4916-baab-5aacadbab524-kserve-provision-location\") pod \"71b9983a-e665-4916-baab-5aacadbab524\" (UID: \"71b9983a-e665-4916-baab-5aacadbab524\") "
Sep 17 18:08:28.360421 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.360379    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71b9983a-e665-4916-baab-5aacadbab524" (UID: "71b9983a-e665-4916-baab-5aacadbab524"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:28.360486 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.360432    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71b9983a-e665-4916-baab-5aacadbab524" (UID: "71b9983a-e665-4916-baab-5aacadbab524"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:28.360546 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.360516    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71b9983a-e665-4916-baab-5aacadbab524-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71b9983a-e665-4916-baab-5aacadbab524" (UID: "71b9983a-e665-4916-baab-5aacadbab524"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:28.362110 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.362085    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b9983a-e665-4916-baab-5aacadbab524-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71b9983a-e665-4916-baab-5aacadbab524" (UID: "71b9983a-e665-4916-baab-5aacadbab524"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:28.362224 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.362126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b9983a-e665-4916-baab-5aacadbab524-kube-api-access-kbjmw" (OuterVolumeSpecName: "kube-api-access-kbjmw") pod "71b9983a-e665-4916-baab-5aacadbab524" (UID: "71b9983a-e665-4916-baab-5aacadbab524"). InnerVolumeSpecName "kube-api-access-kbjmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:28.461324 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.461290    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbjmw\" (UniqueName: \"kubernetes.io/projected/71b9983a-e665-4916-baab-5aacadbab524-kube-api-access-kbjmw\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.461324 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.461317    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.461324 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.461327    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71b9983a-e665-4916-baab-5aacadbab524-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.461539 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.461335    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71b9983a-e665-4916-baab-5aacadbab524-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.461539 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.461345    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71b9983a-e665-4916-baab-5aacadbab524-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:28.633590 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.633513    2562 generic.go:356] "Generic (PLEG): container finished" podID="71b9983a-e665-4916-baab-5aacadbab524" containerID="9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db" exitCode=0
Sep 17 18:08:28.633590 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.633575    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"
Sep 17 18:08:28.633802 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.633591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerDied","Data":"9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db"}
Sep 17 18:08:28.633802 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.633650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6" event={"ID":"71b9983a-e665-4916-baab-5aacadbab524","Type":"ContainerDied","Data":"5fd28e2f2af85b281721b93f467399d14521827f68a0c0bf0d12ad1e11cf7c38"}
Sep 17 18:08:28.633802 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.633678    2562 scope.go:117] "RemoveContainer" containerID="f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1"
Sep 17 18:08:28.642139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.642124    2562 scope.go:117] "RemoveContainer" containerID="9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db"
Sep 17 18:08:28.648859 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.648843    2562 scope.go:117] "RemoveContainer" containerID="7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48"
Sep 17 18:08:28.655818 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.655802    2562 scope.go:117] "RemoveContainer" containerID="f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1"
Sep 17 18:08:28.656114 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:28.656095    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1\": container with ID starting with f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1 not found: ID does not exist" containerID="f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1"
Sep 17 18:08:28.656171 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.656122    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1"} err="failed to get container status \"f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1\": rpc error: code = NotFound desc = could not find container \"f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1\": container with ID starting with f6eecba39a8416a71d51cf916431102dbf13bfcdab3d1615887d2046752d9ac1 not found: ID does not exist"
Sep 17 18:08:28.656171 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.656139    2562 scope.go:117] "RemoveContainer" containerID="9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db"
Sep 17 18:08:28.656387 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:28.656370    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db\": container with ID starting with 9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db not found: ID does not exist" containerID="9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db"
Sep 17 18:08:28.656445 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.656393    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db"} err="failed to get container status \"9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db\": rpc error: code = NotFound desc = could not find container \"9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db\": container with ID starting with 9b3a1111d582b8b47d025a8f51669fd905fe1da69ac9f8acb942968d2ab8a6db not found: ID does not exist"
Sep 17 18:08:28.656445 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.656410    2562 scope.go:117] "RemoveContainer" containerID="7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48"
Sep 17 18:08:28.656628 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:28.656611    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48\": container with ID starting with 7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48 not found: ID does not exist" containerID="7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48"
Sep 17 18:08:28.656681 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.656640    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48"} err="failed to get container status \"7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48\": rpc error: code = NotFound desc = could not find container \"7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48\": container with ID starting with 7e245bbd7147b667f2c0929f35cc7712772807345977c4676fb4627b3078da48 not found: ID does not exist"
Sep 17 18:08:28.665799 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.665779    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"]
Sep 17 18:08:28.671920 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.671887    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-b6f7cbdf5-qnsd6"]
Sep 17 18:08:28.829816 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.829784    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"]
Sep 17 18:08:28.830080 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830065    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="storage-initializer"
Sep 17 18:08:28.830136 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830082    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="storage-initializer"
Sep 17 18:08:28.830136 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830092    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kube-rbac-proxy"
Sep 17 18:08:28.830136 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830098    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kube-rbac-proxy"
Sep 17 18:08:28.830136 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830108    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container"
Sep 17 18:08:28.830136 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830113    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container"
Sep 17 18:08:28.830136 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830125    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container"
Sep 17 18:08:28.830136 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830132    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830141    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kube-rbac-proxy"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830148    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kube-rbac-proxy"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830159    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="storage-initializer"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830165    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="storage-initializer"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830202    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kube-rbac-proxy"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830210    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b9983a-e665-4916-baab-5aacadbab524" containerName="kserve-container"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830219    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kserve-container"
Sep 17 18:08:28.830461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.830229    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4bb6598-c5cc-4ce3-ac27-54f2f5c4dbc0" containerName="kube-rbac-proxy"
Sep 17 18:08:28.835311 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.835283    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:28.837786 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.837763    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:08:28.837903 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.837888    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:08:28.848424 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.848399    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"]
Sep 17 18:08:28.964998 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.964897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:28.964998 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.964966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:28.964998 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.964991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:28.965202 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.965018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45vc\" (UniqueName: \"kubernetes.io/projected/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-api-access-k45vc\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:28.965202 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:28.965048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.065697 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.065662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.065697 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.065699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.065904 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.065721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.065904 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.065755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45vc\" (UniqueName: \"kubernetes.io/projected/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-api-access-k45vc\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.065904 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.065803    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.066151 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.066131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.066416 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.066386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.066516 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.066452    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.068441 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.068419    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-proxy-tls\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.074545 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.074524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45vc\" (UniqueName: \"kubernetes.io/projected/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-api-access-k45vc\") pod \"isvc-canary-promote-predictor-5fb5f89d9d-xbhft\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.144819 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.144790    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:29.276251 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.276223    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b9983a-e665-4916-baab-5aacadbab524" path="/var/lib/kubelet/pods/71b9983a-e665-4916-baab-5aacadbab524/volumes"
Sep 17 18:08:29.416476 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.416391    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"]
Sep 17 18:08:29.419035 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:08:29.419012    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423ca80f_28c5_44a8_afbe_ed3ef9bd3aec.slice/crio-bafd6ab6f040c9ee0469a683cb9384ace158445ed7777261768bdfd979a8bd0d WatchSource:0}: Error finding container bafd6ab6f040c9ee0469a683cb9384ace158445ed7777261768bdfd979a8bd0d: Status 404 returned error can't find the container with id bafd6ab6f040c9ee0469a683cb9384ace158445ed7777261768bdfd979a8bd0d
Sep 17 18:08:29.639118 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.639024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerStarted","Data":"7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a"}
Sep 17 18:08:29.639118 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:29.639106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerStarted","Data":"bafd6ab6f040c9ee0469a683cb9384ace158445ed7777261768bdfd979a8bd0d"}
Sep 17 18:08:30.361081 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:30.361050    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:08:33.655070 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:33.654988    2562 generic.go:356] "Generic (PLEG): container finished" podID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerID="7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a" exitCode=0
Sep 17 18:08:33.655070 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:33.655055    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerDied","Data":"7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a"}
Sep 17 18:08:34.661008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:34.660971    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerStarted","Data":"fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419"}
Sep 17 18:08:34.661375 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:34.661058    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerStarted","Data":"34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c"}
Sep 17 18:08:34.661375 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:34.661330    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:34.661501 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:34.661390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:34.662699 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:34.662674    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:08:34.682419 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:34.682373    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podStartSLOduration=2.786896797 podStartE2EDuration="6.682357559s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.895460762s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:34.680779024 +0000 UTC m=+2005.868214873" watchObservedRunningTime="2026-09-17 18:08:34.682357559 +0000 UTC m=+2005.869793408"
Sep 17 18:08:35.665210 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:35.665165    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:08:37.683616 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:37.683582    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"]
Sep 17 18:08:37.684010 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:37.683864    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" containerID="cri-o://10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83" gracePeriod=30
Sep 17 18:08:37.684010 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:37.683889    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kube-rbac-proxy" containerID="cri-o://f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151" gracePeriod=30
Sep 17 18:08:38.678964 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:38.678929    2562 generic.go:356] "Generic (PLEG): container finished" podID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerID="f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151" exitCode=2
Sep 17 18:08:38.679142 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:38.678985    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerDied","Data":"f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151"}
Sep 17 18:08:40.355483 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:40.355440    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 17 18:08:40.360795 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:40.360770    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 17 18:08:40.669837 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:40.669753    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:08:40.670336 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:40.670310    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:08:42.422880 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.422855    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:08:42.571110 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571071    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vljp4\" (UniqueName: \"kubernetes.io/projected/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-api-access-vljp4\") pod \"173303a9-982c-4524-93ef-b8d47bfa27d4\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") "
Sep 17 18:08:42.571270 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173303a9-982c-4524-93ef-b8d47bfa27d4-kserve-provision-location\") pod \"173303a9-982c-4524-93ef-b8d47bfa27d4\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") "
Sep 17 18:08:42.571270 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571224    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173303a9-982c-4524-93ef-b8d47bfa27d4-proxy-tls\") pod \"173303a9-982c-4524-93ef-b8d47bfa27d4\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") "
Sep 17 18:08:42.571270 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-rbac-proxy-sar-config\") pod \"173303a9-982c-4524-93ef-b8d47bfa27d4\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") "
Sep 17 18:08:42.571465 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571285    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-cabundle-cert\") pod \"173303a9-982c-4524-93ef-b8d47bfa27d4\" (UID: \"173303a9-982c-4524-93ef-b8d47bfa27d4\") "
Sep 17 18:08:42.571595 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571550    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/173303a9-982c-4524-93ef-b8d47bfa27d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "173303a9-982c-4524-93ef-b8d47bfa27d4" (UID: "173303a9-982c-4524-93ef-b8d47bfa27d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:08:42.571721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571650    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "173303a9-982c-4524-93ef-b8d47bfa27d4" (UID: "173303a9-982c-4524-93ef-b8d47bfa27d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:42.571721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.571693    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "173303a9-982c-4524-93ef-b8d47bfa27d4" (UID: "173303a9-982c-4524-93ef-b8d47bfa27d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:08:42.573353 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.573327    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173303a9-982c-4524-93ef-b8d47bfa27d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "173303a9-982c-4524-93ef-b8d47bfa27d4" (UID: "173303a9-982c-4524-93ef-b8d47bfa27d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:08:42.573467 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.573367    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-api-access-vljp4" (OuterVolumeSpecName: "kube-api-access-vljp4") pod "173303a9-982c-4524-93ef-b8d47bfa27d4" (UID: "173303a9-982c-4524-93ef-b8d47bfa27d4"). InnerVolumeSpecName "kube-api-access-vljp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:08:42.671961 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.671926    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/173303a9-982c-4524-93ef-b8d47bfa27d4-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:42.671961 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.671955    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/173303a9-982c-4524-93ef-b8d47bfa27d4-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:42.671961 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.671964    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:42.672183 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.671973    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/173303a9-982c-4524-93ef-b8d47bfa27d4-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:42.672183 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.671982    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vljp4\" (UniqueName: \"kubernetes.io/projected/173303a9-982c-4524-93ef-b8d47bfa27d4-kube-api-access-vljp4\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:08:42.696022 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.695991    2562 generic.go:356] "Generic (PLEG): container finished" podID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerID="10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83" exitCode=0
Sep 17 18:08:42.696168 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.696054    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"
Sep 17 18:08:42.696168 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.696074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerDied","Data":"10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83"}
Sep 17 18:08:42.696168 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.696133    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2" event={"ID":"173303a9-982c-4524-93ef-b8d47bfa27d4","Type":"ContainerDied","Data":"4fb45e0947b65170f0a20aafc14888deb919f21d97d162e0f1e89bd4d281b70d"}
Sep 17 18:08:42.696168 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.696160    2562 scope.go:117] "RemoveContainer" containerID="f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151"
Sep 17 18:08:42.705458 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.705438    2562 scope.go:117] "RemoveContainer" containerID="10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83"
Sep 17 18:08:42.712697 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.712683    2562 scope.go:117] "RemoveContainer" containerID="84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea"
Sep 17 18:08:42.720250 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.720060    2562 scope.go:117] "RemoveContainer" containerID="f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151"
Sep 17 18:08:42.720391 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:42.720371    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151\": container with ID starting with f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151 not found: ID does not exist" containerID="f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151"
Sep 17 18:08:42.720459 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.720399    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151"} err="failed to get container status \"f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151\": rpc error: code = NotFound desc = could not find container \"f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151\": container with ID starting with f5decc3f4c4507ed2eb63975800035201e2511832100c5652b36f297e4cf0151 not found: ID does not exist"
Sep 17 18:08:42.720459 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.720417    2562 scope.go:117] "RemoveContainer" containerID="10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83"
Sep 17 18:08:42.720675 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:42.720656    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83\": container with ID starting with 10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83 not found: ID does not exist" containerID="10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83"
Sep 17 18:08:42.720733 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.720680    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83"} err="failed to get container status \"10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83\": rpc error: code = NotFound desc = could not find container \"10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83\": container with ID starting with 10bbf645fce9b7c021f751834e99dd1f42880d424d0e98646087a987aee71d83 not found: ID does not exist"
Sep 17 18:08:42.720733 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.720696    2562 scope.go:117] "RemoveContainer" containerID="84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea"
Sep 17 18:08:42.720821 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.720740    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"]
Sep 17 18:08:42.720966 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:08:42.720952    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea\": container with ID starting with 84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea not found: ID does not exist" containerID="84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea"
Sep 17 18:08:42.721011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.720970    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea"} err="failed to get container status \"84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea\": rpc error: code = NotFound desc = could not find container \"84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea\": container with ID starting with 84a1de3022917a10abd303b3dcc027dc01e3db19c2a069ecb68a58350ea3eeea not found: ID does not exist"
Sep 17 18:08:42.725327 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.725305    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-666d948758-rf8z2"]
Sep 17 18:08:42.870785 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.870707    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"]
Sep 17 18:08:42.871004 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.870990    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container"
Sep 17 18:08:42.871046 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.871007    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container"
Sep 17 18:08:42.871046 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.871015    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kube-rbac-proxy"
Sep 17 18:08:42.871046 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.871024    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kube-rbac-proxy"
Sep 17 18:08:42.871046 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.871035    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="storage-initializer"
Sep 17 18:08:42.871046 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.871041    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="storage-initializer"
Sep 17 18:08:42.871208 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.871101    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kube-rbac-proxy"
Sep 17 18:08:42.871208 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.871114    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" containerName="kserve-container"
Sep 17 18:08:42.875429 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.875408    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:42.877814 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.877789    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 18:08:42.878195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.878178    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 18:08:42.888062 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.888039    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"]
Sep 17 18:08:42.974356 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.974325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6647caac-1f00-4481-8daa-b1d74683f22d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:42.974356 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.974359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6647caac-1f00-4481-8daa-b1d74683f22d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:42.974593 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.974388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:42.974593 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.974494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:42.974593 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:42.974573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh4m4\" (UniqueName: \"kubernetes.io/projected/6647caac-1f00-4481-8daa-b1d74683f22d-kube-api-access-kh4m4\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075208 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075208 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh4m4\" (UniqueName: \"kubernetes.io/projected/6647caac-1f00-4481-8daa-b1d74683f22d-kube-api-access-kh4m4\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6647caac-1f00-4481-8daa-b1d74683f22d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6647caac-1f00-4481-8daa-b1d74683f22d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075740 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075717    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6647caac-1f00-4481-8daa-b1d74683f22d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075934 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.075934 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.075925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.077746 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.077729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6647caac-1f00-4481-8daa-b1d74683f22d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.091299 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.091274    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh4m4\" (UniqueName: \"kubernetes.io/projected/6647caac-1f00-4481-8daa-b1d74683f22d-kube-api-access-kh4m4\") pod \"isvc-tensorflow-predictor-547dfc5669-v8b28\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.185084 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.185004    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:43.276432 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.276393    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="173303a9-982c-4524-93ef-b8d47bfa27d4" path="/var/lib/kubelet/pods/173303a9-982c-4524-93ef-b8d47bfa27d4/volumes"
Sep 17 18:08:43.473763 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.473738    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"]
Sep 17 18:08:43.476138 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:08:43.476102    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6647caac_1f00_4481_8daa_b1d74683f22d.slice/crio-5933de0609f67b90cb77516e9c75130a90e92fcadfa11e7ec08ca487121a55ea WatchSource:0}: Error finding container 5933de0609f67b90cb77516e9c75130a90e92fcadfa11e7ec08ca487121a55ea: Status 404 returned error can't find the container with id 5933de0609f67b90cb77516e9c75130a90e92fcadfa11e7ec08ca487121a55ea
Sep 17 18:08:43.702492 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.702457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerStarted","Data":"e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15"}
Sep 17 18:08:43.702688 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:43.702524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerStarted","Data":"5933de0609f67b90cb77516e9c75130a90e92fcadfa11e7ec08ca487121a55ea"}
Sep 17 18:08:48.722128 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:48.722096    2562 generic.go:356] "Generic (PLEG): container finished" podID="6647caac-1f00-4481-8daa-b1d74683f22d" containerID="e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15" exitCode=0
Sep 17 18:08:48.722538 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:48.722172    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerDied","Data":"e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15"}
Sep 17 18:08:48.723354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:48.723339    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:08:50.671031 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:50.670990    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:08:52.741637 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:52.741598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerStarted","Data":"94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f"}
Sep 17 18:08:52.742074 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:52.741690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerStarted","Data":"aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda"}
Sep 17 18:08:52.742074 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:52.742035    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:52.765168 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:52.765111    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podStartSLOduration=2.903887589 podStartE2EDuration="10.765096643s" totalImagesPullingTime="3.497150337s" totalInitContainerRuntime="4.364058717s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:52.764019006 +0000 UTC m=+2023.951454856" watchObservedRunningTime="2026-09-17 18:08:52.765096643 +0000 UTC m=+2023.952532491"
Sep 17 18:08:53.745781 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:53.745741    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:53.746991 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:53.746962    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:08:54.749891 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:54.749851    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:08:59.755281 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:59.755249    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:08:59.755751 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:08:59.755706    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 18:09:00.670357 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:00.670314    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:09:09.756740 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:09.756713    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:09:10.671239 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:10.671200    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:09:20.670797 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:20.670755    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:09:24.077317 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:24.077268    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"]
Sep 17 18:09:24.077821 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:24.077556    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kserve-container" containerID="cri-o://aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda" gracePeriod=30
Sep 17 18:09:24.077821 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:24.077603    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy" containerID="cri-o://94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f" gracePeriod=30
Sep 17 18:09:24.750718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:24.750676    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 18:09:24.866518 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:24.866488    2562 generic.go:356] "Generic (PLEG): container finished" podID="6647caac-1f00-4481-8daa-b1d74683f22d" containerID="94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f" exitCode=2
Sep 17 18:09:24.866694 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:24.866561    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerDied","Data":"94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f"}
Sep 17 18:09:29.751057 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:29.750966    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 18:09:30.670398 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:30.670353    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:09:34.751191 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:34.751145    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 18:09:34.751594 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:34.751283    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:09:39.751040 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:39.750987    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 18:09:40.671127 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:40.671086    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:09:44.750288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:44.750245    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 18:09:48.271860 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:48.271812    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 18:09:49.750238 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:49.750196    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 18:09:53.140045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:53.140006    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"]
Sep 17 18:09:53.140521 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:53.140353    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container" containerID="cri-o://34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c" gracePeriod=30
Sep 17 18:09:53.140521 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:53.140380    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kube-rbac-proxy" containerID="cri-o://fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419" gracePeriod=30
Sep 17 18:09:53.141195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:53.141090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:09:53.980677 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:53.980647    2562 generic.go:356] "Generic (PLEG): container finished" podID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerID="fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419" exitCode=2
Sep 17 18:09:53.980677 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:53.980680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerDied","Data":"fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419"}
Sep 17 18:09:54.713736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.713705    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:09:54.813793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.813743    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-cabundle-cert\") pod \"6647caac-1f00-4481-8daa-b1d74683f22d\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") "
Sep 17 18:09:54.813793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.813798    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-kube-rbac-proxy-sar-config\") pod \"6647caac-1f00-4481-8daa-b1d74683f22d\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") "
Sep 17 18:09:54.814167 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.813848    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6647caac-1f00-4481-8daa-b1d74683f22d-kserve-provision-location\") pod \"6647caac-1f00-4481-8daa-b1d74683f22d\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") "
Sep 17 18:09:54.814167 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.813873    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh4m4\" (UniqueName: \"kubernetes.io/projected/6647caac-1f00-4481-8daa-b1d74683f22d-kube-api-access-kh4m4\") pod \"6647caac-1f00-4481-8daa-b1d74683f22d\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") "
Sep 17 18:09:54.814167 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.813946    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6647caac-1f00-4481-8daa-b1d74683f22d-proxy-tls\") pod \"6647caac-1f00-4481-8daa-b1d74683f22d\" (UID: \"6647caac-1f00-4481-8daa-b1d74683f22d\") "
Sep 17 18:09:54.814278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.814222    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6647caac-1f00-4481-8daa-b1d74683f22d" (UID: "6647caac-1f00-4481-8daa-b1d74683f22d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:54.814278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.814243    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6647caac-1f00-4481-8daa-b1d74683f22d" (UID: "6647caac-1f00-4481-8daa-b1d74683f22d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:54.816161 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.816131    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6647caac-1f00-4481-8daa-b1d74683f22d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6647caac-1f00-4481-8daa-b1d74683f22d" (UID: "6647caac-1f00-4481-8daa-b1d74683f22d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:54.816270 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.816222    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6647caac-1f00-4481-8daa-b1d74683f22d-kube-api-access-kh4m4" (OuterVolumeSpecName: "kube-api-access-kh4m4") pod "6647caac-1f00-4481-8daa-b1d74683f22d" (UID: "6647caac-1f00-4481-8daa-b1d74683f22d"). InnerVolumeSpecName "kube-api-access-kh4m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:54.824699 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.824672    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6647caac-1f00-4481-8daa-b1d74683f22d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6647caac-1f00-4481-8daa-b1d74683f22d" (UID: "6647caac-1f00-4481-8daa-b1d74683f22d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:54.914793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.914697    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:54.914793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.914742    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6647caac-1f00-4481-8daa-b1d74683f22d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:54.914793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.914752    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6647caac-1f00-4481-8daa-b1d74683f22d-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:54.914793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.914761    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh4m4\" (UniqueName: \"kubernetes.io/projected/6647caac-1f00-4481-8daa-b1d74683f22d-kube-api-access-kh4m4\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:54.914793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.914771    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6647caac-1f00-4481-8daa-b1d74683f22d-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:54.986263 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.986229    2562 generic.go:356] "Generic (PLEG): container finished" podID="6647caac-1f00-4481-8daa-b1d74683f22d" containerID="aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda" exitCode=137
Sep 17 18:09:54.986423 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.986286    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"
Sep 17 18:09:54.986423 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.986291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerDied","Data":"aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda"}
Sep 17 18:09:54.986423 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.986403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28" event={"ID":"6647caac-1f00-4481-8daa-b1d74683f22d","Type":"ContainerDied","Data":"5933de0609f67b90cb77516e9c75130a90e92fcadfa11e7ec08ca487121a55ea"}
Sep 17 18:09:54.986531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.986431    2562 scope.go:117] "RemoveContainer" containerID="94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f"
Sep 17 18:09:54.995041 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:54.995024    2562 scope.go:117] "RemoveContainer" containerID="aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda"
Sep 17 18:09:55.001876 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.001853    2562 scope.go:117] "RemoveContainer" containerID="e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15"
Sep 17 18:09:55.007798 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.007777    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"]
Sep 17 18:09:55.009211 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.009198    2562 scope.go:117] "RemoveContainer" containerID="94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f"
Sep 17 18:09:55.009481 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:09:55.009455    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f\": container with ID starting with 94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f not found: ID does not exist" containerID="94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f"
Sep 17 18:09:55.009567 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.009486    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f"} err="failed to get container status \"94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f\": rpc error: code = NotFound desc = could not find container \"94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f\": container with ID starting with 94245c2ec48ed21283d50b9f813deca00ed173c018b205382f5313a91484683f not found: ID does not exist"
Sep 17 18:09:55.009567 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.009503    2562 scope.go:117] "RemoveContainer" containerID="aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda"
Sep 17 18:09:55.009731 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:09:55.009713    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda\": container with ID starting with aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda not found: ID does not exist" containerID="aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda"
Sep 17 18:09:55.009772 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.009739    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda"} err="failed to get container status \"aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda\": rpc error: code = NotFound desc = could not find container \"aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda\": container with ID starting with aab8bfb1c598b0880e88156a0fe96b5d616c47780d1e3b58db353317fd216bda not found: ID does not exist"
Sep 17 18:09:55.009772 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.009757    2562 scope.go:117] "RemoveContainer" containerID="e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15"
Sep 17 18:09:55.010026 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:09:55.010002    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15\": container with ID starting with e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15 not found: ID does not exist" containerID="e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15"
Sep 17 18:09:55.010077 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.010025    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15"} err="failed to get container status \"e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15\": rpc error: code = NotFound desc = could not find container \"e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15\": container with ID starting with e55f84ac7ff7e8acc3263e619ccb1fed43353cd511faa2d2a20b20395a349e15 not found: ID does not exist"
Sep 17 18:09:55.013945 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.013924    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-v8b28"]
Sep 17 18:09:55.276045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.276011    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" path="/var/lib/kubelet/pods/6647caac-1f00-4481-8daa-b1d74683f22d/volumes"
Sep 17 18:09:55.666174 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:55.666079    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 18:09:57.977719 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:57.977696    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:09:57.999666 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:57.999636    2562 generic.go:356] "Generic (PLEG): container finished" podID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerID="34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c" exitCode=0
Sep 17 18:09:57.999787 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:57.999687    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"
Sep 17 18:09:57.999787 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:57.999691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerDied","Data":"34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c"}
Sep 17 18:09:57.999963 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:57.999810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft" event={"ID":"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec","Type":"ContainerDied","Data":"bafd6ab6f040c9ee0469a683cb9384ace158445ed7777261768bdfd979a8bd0d"}
Sep 17 18:09:57.999963 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:57.999839    2562 scope.go:117] "RemoveContainer" containerID="fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419"
Sep 17 18:09:58.007679 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.007660    2562 scope.go:117] "RemoveContainer" containerID="34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c"
Sep 17 18:09:58.016353 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.016330    2562 scope.go:117] "RemoveContainer" containerID="7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a"
Sep 17 18:09:58.023127 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.023103    2562 scope.go:117] "RemoveContainer" containerID="fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419"
Sep 17 18:09:58.023396 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:09:58.023376    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419\": container with ID starting with fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419 not found: ID does not exist" containerID="fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419"
Sep 17 18:09:58.023447 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.023409    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419"} err="failed to get container status \"fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419\": rpc error: code = NotFound desc = could not find container \"fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419\": container with ID starting with fe05f9a70b0deec14ccd5e2bff32d9f5d3fc513e154dc81c4d0491d66f72c419 not found: ID does not exist"
Sep 17 18:09:58.023447 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.023428    2562 scope.go:117] "RemoveContainer" containerID="34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c"
Sep 17 18:09:58.023670 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:09:58.023652    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c\": container with ID starting with 34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c not found: ID does not exist" containerID="34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c"
Sep 17 18:09:58.023736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.023678    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c"} err="failed to get container status \"34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c\": rpc error: code = NotFound desc = could not find container \"34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c\": container with ID starting with 34ac357bdabd5eb32992d97487be1cbcfe5e7e1575f3b5caec306986e558724c not found: ID does not exist"
Sep 17 18:09:58.023736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.023700    2562 scope.go:117] "RemoveContainer" containerID="7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a"
Sep 17 18:09:58.024002 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:09:58.023986    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a\": container with ID starting with 7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a not found: ID does not exist" containerID="7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a"
Sep 17 18:09:58.024044 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.024004    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a"} err="failed to get container status \"7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a\": rpc error: code = NotFound desc = could not find container \"7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a\": container with ID starting with 7debf981c051ec6548d4b0c6888fcfa489d1a2b8ab958e55e10bc0ac60c1594a not found: ID does not exist"
Sep 17 18:09:58.038298 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.038119    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kserve-provision-location\") pod \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") "
Sep 17 18:09:58.038298 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.038166    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k45vc\" (UniqueName: \"kubernetes.io/projected/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-api-access-k45vc\") pod \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") "
Sep 17 18:09:58.038298 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.038221    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-proxy-tls\") pod \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") "
Sep 17 18:09:58.038298 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.038245    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-rbac-proxy-sar-config\") pod \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") "
Sep 17 18:09:58.038298 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.038274    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-cabundle-cert\") pod \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\" (UID: \"423ca80f-28c5-44a8-afbe-ed3ef9bd3aec\") "
Sep 17 18:09:58.038722 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.038675    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" (UID: "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:58.038788 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.038767    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" (UID: "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:58.039131 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.039086    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" (UID: "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:58.040754 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.040733    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" (UID: "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:58.040832 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.040772    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-api-access-k45vc" (OuterVolumeSpecName: "kube-api-access-k45vc") pod "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" (UID: "423ca80f-28c5-44a8-afbe-ed3ef9bd3aec"). InnerVolumeSpecName "kube-api-access-k45vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:58.139782 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.139744    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:58.139782 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.139773    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k45vc\" (UniqueName: \"kubernetes.io/projected/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-api-access-k45vc\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:58.139782 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.139787    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:58.140039 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.139797    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:58.140039 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.139806    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:09:58.322529 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.322497    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"]
Sep 17 18:09:58.326825 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.326797    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fb5f89d9d-xbhft"]
Sep 17 18:09:58.909832 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.909797    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"]
Sep 17 18:09:58.910072 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910058    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910074    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910084    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="storage-initializer"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910090    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="storage-initializer"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910096    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kserve-container"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910102    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kserve-container"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910116    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kube-rbac-proxy"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910122    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kube-rbac-proxy"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910129    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="storage-initializer"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910134    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="storage-initializer"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910141    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container"
Sep 17 18:09:58.910149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910146    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container"
Sep 17 18:09:58.913354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910846    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kube-rbac-proxy"
Sep 17 18:09:58.913354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910882    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" containerName="kserve-container"
Sep 17 18:09:58.913354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910898    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kube-rbac-proxy"
Sep 17 18:09:58.913354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.910934    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6647caac-1f00-4481-8daa-b1d74683f22d" containerName="kserve-container"
Sep 17 18:09:58.916460 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.916429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:58.918668 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.918631    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:09:58.918847 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.918733    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:09:58.919476 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.919459    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:09:58.919581 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.919500    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:09:58.919581 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.919523    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:09:58.919687 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.919618    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:09:58.919761 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.919683    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:09:58.925090 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:58.925029    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"]
Sep 17 18:09:59.047800 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.047761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe896754-7844-4648-95ca-1b94c5902145-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.047800 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.047800    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.048254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.047829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrpzw\" (UniqueName: \"kubernetes.io/projected/fe896754-7844-4648-95ca-1b94c5902145-kube-api-access-zrpzw\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.048254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.047859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.048254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.047887    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe896754-7844-4648-95ca-1b94c5902145-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149184 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrpzw\" (UniqueName: \"kubernetes.io/projected/fe896754-7844-4648-95ca-1b94c5902145-kube-api-access-zrpzw\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149184 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149420 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149212    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe896754-7844-4648-95ca-1b94c5902145-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149420 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe896754-7844-4648-95ca-1b94c5902145-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149420 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149698 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe896754-7844-4648-95ca-1b94c5902145-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149885 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.149961 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.149876    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.151716 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.151696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe896754-7844-4648-95ca-1b94c5902145-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.159206 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.159179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrpzw\" (UniqueName: \"kubernetes.io/projected/fe896754-7844-4648-95ca-1b94c5902145-kube-api-access-zrpzw\") pod \"isvc-canary-promote-v2-predictor-584d7dc949-rxm4v\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.228443 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.228347    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:09:59.277380 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.277130    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423ca80f-28c5-44a8-afbe-ed3ef9bd3aec" path="/var/lib/kubelet/pods/423ca80f-28c5-44a8-afbe-ed3ef9bd3aec/volumes"
Sep 17 18:09:59.503894 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:09:59.503862    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"]
Sep 17 18:09:59.507343 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:09:59.507312    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe896754_7844_4648_95ca_1b94c5902145.slice/crio-12f5dcafdb0872a5bfecbafdaabde84b9dc6dba2d2fc0ca95a8c3795bc09d943 WatchSource:0}: Error finding container 12f5dcafdb0872a5bfecbafdaabde84b9dc6dba2d2fc0ca95a8c3795bc09d943: Status 404 returned error can't find the container with id 12f5dcafdb0872a5bfecbafdaabde84b9dc6dba2d2fc0ca95a8c3795bc09d943
Sep 17 18:10:00.009646 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:00.009609    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerStarted","Data":"9fdbf1411f910d2df981b414bdcde5e993dd67ed1b7e3b4e344d99b8b5049843"}
Sep 17 18:10:00.009821 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:00.009693    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerStarted","Data":"12f5dcafdb0872a5bfecbafdaabde84b9dc6dba2d2fc0ca95a8c3795bc09d943"}
Sep 17 18:10:05.027358 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:05.027324    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe896754-7844-4648-95ca-1b94c5902145" containerID="9fdbf1411f910d2df981b414bdcde5e993dd67ed1b7e3b4e344d99b8b5049843" exitCode=0
Sep 17 18:10:05.027732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:05.027382    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerDied","Data":"9fdbf1411f910d2df981b414bdcde5e993dd67ed1b7e3b4e344d99b8b5049843"}
Sep 17 18:10:06.033097 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:06.033062    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerStarted","Data":"7fd11e005083d2309c3386b8ac9346c0fab36e686a277fd6185986552109d982"}
Sep 17 18:10:06.033482 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:06.033149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerStarted","Data":"dba0599047bb0193b4e3c0a33d6e69f5a4796248bdd14ae0f5a1ab8e90d7df53"}
Sep 17 18:10:06.033482 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:06.033388    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:10:06.056775 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:06.056723    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podStartSLOduration=2.787501464 podStartE2EDuration="8.05670713s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.269205666s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:06.055675678 +0000 UTC m=+2097.243111527" watchObservedRunningTime="2026-09-17 18:10:06.05670713 +0000 UTC m=+2097.244143043"
Sep 17 18:10:07.037100 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:07.037067    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:10:07.038278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:07.038249    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:10:08.041077 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:08.041045    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:10:13.045588 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:13.045558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:10:13.046116 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:13.046091    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:10:16.067063 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.067032    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"]
Sep 17 18:10:16.070219 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.070200    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.072635 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.072611    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 18:10:16.072732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.072616    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 18:10:16.082000 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.081977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"]
Sep 17 18:10:16.183585 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.183548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.183585 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.183597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff8a33b-c9ef-491c-a082-027e9cd21b0b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.183858 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.183616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.183858 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.183725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjmb\" (UniqueName: \"kubernetes.io/projected/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-api-access-4hjmb\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.183858 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.183771    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.284805 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.284764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff8a33b-c9ef-491c-a082-027e9cd21b0b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.284805 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.284802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.285047 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.284847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjmb\" (UniqueName: \"kubernetes.io/projected/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-api-access-4hjmb\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.285047 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.284883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.285047 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.284948    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.285422 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.285399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.285690 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.285669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.285735 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.285668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.287357 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.287338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff8a33b-c9ef-491c-a082-027e9cd21b0b-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.293351 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.293327    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjmb\" (UniqueName: \"kubernetes.io/projected/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-api-access-4hjmb\") pod \"isvc-triton-predictor-5d7db58676-jxckg\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.380450 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.380352    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:10:16.651891 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:16.651843    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"]
Sep 17 18:10:16.655999 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:10:16.655965    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff8a33b_c9ef_491c_a082_027e9cd21b0b.slice/crio-91c3b2a114b7fbfc0c5aa7ab6289d7712c206b9f74e9df22bddf6716fd438b98 WatchSource:0}: Error finding container 91c3b2a114b7fbfc0c5aa7ab6289d7712c206b9f74e9df22bddf6716fd438b98: Status 404 returned error can't find the container with id 91c3b2a114b7fbfc0c5aa7ab6289d7712c206b9f74e9df22bddf6716fd438b98
Sep 17 18:10:17.074077 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:17.074035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerStarted","Data":"91bdc7013705e7f8cf5956941178e4c71e9cba245dbe2d4dac119113e7490d6d"}
Sep 17 18:10:17.074482 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:17.074111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerStarted","Data":"91c3b2a114b7fbfc0c5aa7ab6289d7712c206b9f74e9df22bddf6716fd438b98"}
Sep 17 18:10:21.094354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:21.094267    2562 generic.go:356] "Generic (PLEG): container finished" podID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerID="91bdc7013705e7f8cf5956941178e4c71e9cba245dbe2d4dac119113e7490d6d" exitCode=0
Sep 17 18:10:21.094354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:21.094313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerDied","Data":"91bdc7013705e7f8cf5956941178e4c71e9cba245dbe2d4dac119113e7490d6d"}
Sep 17 18:10:23.048180 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:23.047742    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:10:33.047344 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:33.046896    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:10:43.046335 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:43.046294    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:10:53.046964 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:10:53.046875    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:11:03.046246 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:03.046102    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:11:13.047150 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:13.047110    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:11:23.047018 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:23.046977    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:11:32.272384 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:32.272340    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:11:42.272902 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:42.272869    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:11:49.137475 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:49.137441    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"]
Sep 17 18:11:49.137995 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:49.137704    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" containerID="cri-o://dba0599047bb0193b4e3c0a33d6e69f5a4796248bdd14ae0f5a1ab8e90d7df53" gracePeriod=30
Sep 17 18:11:49.137995 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:49.137744    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy" containerID="cri-o://7fd11e005083d2309c3386b8ac9346c0fab36e686a277fd6185986552109d982" gracePeriod=30
Sep 17 18:11:52.272299 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:52.272255    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:11:53.042052 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:53.042008    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 18:11:54.345255 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.345214    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"]
Sep 17 18:11:54.369626 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.369590    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"]
Sep 17 18:11:54.369829 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.369743    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.372454 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.372430    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:11:54.372667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.372648    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:11:54.404648 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.404614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.404827 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.404664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hw5\" (UniqueName: \"kubernetes.io/projected/869453aa-6892-499d-8b34-756eb766a718-kube-api-access-d5hw5\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.404827 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.404743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.404827 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.404797    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/869453aa-6892-499d-8b34-756eb766a718-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.404827 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.404823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/869453aa-6892-499d-8b34-756eb766a718-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.505693 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.505588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.505693 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.505661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/869453aa-6892-499d-8b34-756eb766a718-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.505693 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.505689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/869453aa-6892-499d-8b34-756eb766a718-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.506108 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.505746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.506108 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.505784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hw5\" (UniqueName: \"kubernetes.io/projected/869453aa-6892-499d-8b34-756eb766a718-kube-api-access-d5hw5\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.506217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.506155    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/869453aa-6892-499d-8b34-756eb766a718-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.506426 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.506402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-cabundle-cert\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.506563 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.506538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.509109 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.509085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/869453aa-6892-499d-8b34-756eb766a718-proxy-tls\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.514792 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.514771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hw5\" (UniqueName: \"kubernetes.io/projected/869453aa-6892-499d-8b34-756eb766a718-kube-api-access-d5hw5\") pod \"isvc-canary-rollback-predictor-786544c6c8-ghxsd\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:54.682183 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:54.682141    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:11:58.041630 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:11:58.041574    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 18:12:02.272753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:02.272706    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:12:03.042060 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:03.042010    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 18:12:03.042272 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:03.042157    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:12:08.041600 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:08.041560    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 18:12:12.272235 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:12.272196    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 18:12:12.272708 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:12.272316    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:12:13.041414 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.041361    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 17 18:12:13.620292 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.620260    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe896754-7844-4648-95ca-1b94c5902145" containerID="7fd11e005083d2309c3386b8ac9346c0fab36e686a277fd6185986552109d982" exitCode=2
Sep 17 18:12:13.620292 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.620288    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe896754-7844-4648-95ca-1b94c5902145" containerID="dba0599047bb0193b4e3c0a33d6e69f5a4796248bdd14ae0f5a1ab8e90d7df53" exitCode=0
Sep 17 18:12:13.620713 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.620334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerDied","Data":"7fd11e005083d2309c3386b8ac9346c0fab36e686a277fd6185986552109d982"}
Sep 17 18:12:13.620713 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.620404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerDied","Data":"dba0599047bb0193b4e3c0a33d6e69f5a4796248bdd14ae0f5a1ab8e90d7df53"}
Sep 17 18:12:13.742473 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.742445    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:12:13.872530 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.872497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrpzw\" (UniqueName: \"kubernetes.io/projected/fe896754-7844-4648-95ca-1b94c5902145-kube-api-access-zrpzw\") pod \"fe896754-7844-4648-95ca-1b94c5902145\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") "
Sep 17 18:12:13.872695 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.872561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe896754-7844-4648-95ca-1b94c5902145-proxy-tls\") pod \"fe896754-7844-4648-95ca-1b94c5902145\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") "
Sep 17 18:12:13.872695 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.872630    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-kube-rbac-proxy-sar-config\") pod \"fe896754-7844-4648-95ca-1b94c5902145\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") "
Sep 17 18:12:13.872695 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.872660    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe896754-7844-4648-95ca-1b94c5902145-kserve-provision-location\") pod \"fe896754-7844-4648-95ca-1b94c5902145\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") "
Sep 17 18:12:13.872870 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.872697    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-cabundle-cert\") pod \"fe896754-7844-4648-95ca-1b94c5902145\" (UID: \"fe896754-7844-4648-95ca-1b94c5902145\") "
Sep 17 18:12:13.873488 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.873439    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe896754-7844-4648-95ca-1b94c5902145-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fe896754-7844-4648-95ca-1b94c5902145" (UID: "fe896754-7844-4648-95ca-1b94c5902145"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:13.874000 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.873709    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fe896754-7844-4648-95ca-1b94c5902145" (UID: "fe896754-7844-4648-95ca-1b94c5902145"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:13.874071 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.874011    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fe896754-7844-4648-95ca-1b94c5902145" (UID: "fe896754-7844-4648-95ca-1b94c5902145"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:13.875727 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.875694    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe896754-7844-4648-95ca-1b94c5902145-kube-api-access-zrpzw" (OuterVolumeSpecName: "kube-api-access-zrpzw") pod "fe896754-7844-4648-95ca-1b94c5902145" (UID: "fe896754-7844-4648-95ca-1b94c5902145"). InnerVolumeSpecName "kube-api-access-zrpzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:13.876240 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.876215    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe896754-7844-4648-95ca-1b94c5902145-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe896754-7844-4648-95ca-1b94c5902145" (UID: "fe896754-7844-4648-95ca-1b94c5902145"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:13.973225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.973151    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:13.973225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.973179    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe896754-7844-4648-95ca-1b94c5902145-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:13.973225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.973191    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe896754-7844-4648-95ca-1b94c5902145-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:13.973225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.973203    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrpzw\" (UniqueName: \"kubernetes.io/projected/fe896754-7844-4648-95ca-1b94c5902145-kube-api-access-zrpzw\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:13.973225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:13.973217    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe896754-7844-4648-95ca-1b94c5902145-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:14.097571 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.097528    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"]
Sep 17 18:12:14.099970 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:12:14.099940    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869453aa_6892_499d_8b34_756eb766a718.slice/crio-fdb410e020e045943d498e47d88a9e60379963366d7c365f47fd41dfb99e7721 WatchSource:0}: Error finding container fdb410e020e045943d498e47d88a9e60379963366d7c365f47fd41dfb99e7721: Status 404 returned error can't find the container with id fdb410e020e045943d498e47d88a9e60379963366d7c365f47fd41dfb99e7721
Sep 17 18:12:14.626627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.626546    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"
Sep 17 18:12:14.626627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.626566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v" event={"ID":"fe896754-7844-4648-95ca-1b94c5902145","Type":"ContainerDied","Data":"12f5dcafdb0872a5bfecbafdaabde84b9dc6dba2d2fc0ca95a8c3795bc09d943"}
Sep 17 18:12:14.627175 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.626640    2562 scope.go:117] "RemoveContainer" containerID="7fd11e005083d2309c3386b8ac9346c0fab36e686a277fd6185986552109d982"
Sep 17 18:12:14.628340 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.628313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerStarted","Data":"82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56"}
Sep 17 18:12:14.628456 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.628361    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerStarted","Data":"fdb410e020e045943d498e47d88a9e60379963366d7c365f47fd41dfb99e7721"}
Sep 17 18:12:14.636057 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.635945    2562 scope.go:117] "RemoveContainer" containerID="dba0599047bb0193b4e3c0a33d6e69f5a4796248bdd14ae0f5a1ab8e90d7df53"
Sep 17 18:12:14.643804 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.643778    2562 scope.go:117] "RemoveContainer" containerID="9fdbf1411f910d2df981b414bdcde5e993dd67ed1b7e3b4e344d99b8b5049843"
Sep 17 18:12:14.665967 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.665934    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"]
Sep 17 18:12:14.672926 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:14.672882    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-584d7dc949-rxm4v"]
Sep 17 18:12:15.276250 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:15.276219    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe896754-7844-4648-95ca-1b94c5902145" path="/var/lib/kubelet/pods/fe896754-7844-4648-95ca-1b94c5902145/volumes"
Sep 17 18:12:15.635289 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:15.635195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerStarted","Data":"8ab8dc1992998097ba3553e6d40e54208ab0720ec6650718d480ac7cf77cbafb"}
Sep 17 18:12:15.635289 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:15.635258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerStarted","Data":"596954fad0a196322500865e2449258ae76949fa1d453f28c55e042f149f4205"}
Sep 17 18:12:15.658287 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:15.658236    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" podStartSLOduration=1.815965263 podStartE2EDuration="1m59.658221263s" totalImagesPullingTime="1m53.764546879s" totalInitContainerRuntime="4.077709121s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:15.656663186 +0000 UTC m=+2226.844099035" watchObservedRunningTime="2026-09-17 18:12:15.658221263 +0000 UTC m=+2226.845657112"
Sep 17 18:12:16.638680 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:16.638640    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:12:16.639167 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:16.638720    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:12:16.640057 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:16.640026    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:12:17.642037 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:17.641999    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 18:12:19.651427 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:19.651386    2562 generic.go:356] "Generic (PLEG): container finished" podID="869453aa-6892-499d-8b34-756eb766a718" containerID="82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56" exitCode=0
Sep 17 18:12:19.651799 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:19.651455    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerDied","Data":"82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56"}
Sep 17 18:12:20.658181 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:20.658137    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerStarted","Data":"7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08"}
Sep 17 18:12:20.658642 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:20.658257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerStarted","Data":"db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4"}
Sep 17 18:12:20.658642 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:20.658500    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:12:20.680954 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:20.680877    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podStartSLOduration=22.301598515 podStartE2EDuration="26.680863908s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.379265393s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:20.680312573 +0000 UTC m=+2231.867748422" watchObservedRunningTime="2026-09-17 18:12:20.680863908 +0000 UTC m=+2231.868299816"
Sep 17 18:12:21.662146 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:21.662114    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:12:21.663348 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:21.663322    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:12:22.646976 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:22.646949    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:12:22.647724 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:22.647700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:12:22.666225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:22.666187    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:12:27.670956 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:27.670908    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:12:27.671539 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:27.671510    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:12:28.369462 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:28.369387    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"]
Sep 17 18:12:28.369678 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:28.369654    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kserve-container" containerID="cri-o://596954fad0a196322500865e2449258ae76949fa1d453f28c55e042f149f4205" gracePeriod=30
Sep 17 18:12:28.369751 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:28.369696    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kube-rbac-proxy" containerID="cri-o://8ab8dc1992998097ba3553e6d40e54208ab0720ec6650718d480ac7cf77cbafb" gracePeriod=30
Sep 17 18:12:28.690718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:28.690633    2562 generic.go:356] "Generic (PLEG): container finished" podID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerID="8ab8dc1992998097ba3553e6d40e54208ab0720ec6650718d480ac7cf77cbafb" exitCode=2
Sep 17 18:12:28.690718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:28.690704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerDied","Data":"8ab8dc1992998097ba3553e6d40e54208ab0720ec6650718d480ac7cf77cbafb"}
Sep 17 18:12:30.699863 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.699833    2562 generic.go:356] "Generic (PLEG): container finished" podID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerID="596954fad0a196322500865e2449258ae76949fa1d453f28c55e042f149f4205" exitCode=0
Sep 17 18:12:30.700216 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.699897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerDied","Data":"596954fad0a196322500865e2449258ae76949fa1d453f28c55e042f149f4205"}
Sep 17 18:12:30.723059 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.723036    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:12:30.809259 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kserve-provision-location\") pod \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") "
Sep 17 18:12:30.809259 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809219    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff8a33b-c9ef-491c-a082-027e9cd21b0b-proxy-tls\") pod \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") "
Sep 17 18:12:30.809259 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809241    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-rbac-proxy-sar-config\") pod \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") "
Sep 17 18:12:30.809577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809265    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hjmb\" (UniqueName: \"kubernetes.io/projected/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-api-access-4hjmb\") pod \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") "
Sep 17 18:12:30.809577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809302    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-cabundle-cert\") pod \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\" (UID: \"fff8a33b-c9ef-491c-a082-027e9cd21b0b\") "
Sep 17 18:12:30.809674 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809605    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fff8a33b-c9ef-491c-a082-027e9cd21b0b" (UID: "fff8a33b-c9ef-491c-a082-027e9cd21b0b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:30.809674 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809641    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fff8a33b-c9ef-491c-a082-027e9cd21b0b" (UID: "fff8a33b-c9ef-491c-a082-027e9cd21b0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:30.809744 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.809693    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fff8a33b-c9ef-491c-a082-027e9cd21b0b" (UID: "fff8a33b-c9ef-491c-a082-027e9cd21b0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:30.811352 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.811333    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-api-access-4hjmb" (OuterVolumeSpecName: "kube-api-access-4hjmb") pod "fff8a33b-c9ef-491c-a082-027e9cd21b0b" (UID: "fff8a33b-c9ef-491c-a082-027e9cd21b0b"). InnerVolumeSpecName "kube-api-access-4hjmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:30.811418 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.811403    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff8a33b-c9ef-491c-a082-027e9cd21b0b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fff8a33b-c9ef-491c-a082-027e9cd21b0b" (UID: "fff8a33b-c9ef-491c-a082-027e9cd21b0b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:30.910475 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.910440    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:30.910475 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.910468    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fff8a33b-c9ef-491c-a082-027e9cd21b0b-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:30.910475 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.910478    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:30.910475 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.910486    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjmb\" (UniqueName: \"kubernetes.io/projected/fff8a33b-c9ef-491c-a082-027e9cd21b0b-kube-api-access-4hjmb\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:30.910746 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:30.910495    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fff8a33b-c9ef-491c-a082-027e9cd21b0b-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:12:31.705451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:31.705376    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"
Sep 17 18:12:31.705451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:31.705383    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg" event={"ID":"fff8a33b-c9ef-491c-a082-027e9cd21b0b","Type":"ContainerDied","Data":"91c3b2a114b7fbfc0c5aa7ab6289d7712c206b9f74e9df22bddf6716fd438b98"}
Sep 17 18:12:31.705451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:31.705444    2562 scope.go:117] "RemoveContainer" containerID="8ab8dc1992998097ba3553e6d40e54208ab0720ec6650718d480ac7cf77cbafb"
Sep 17 18:12:31.715395 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:31.715372    2562 scope.go:117] "RemoveContainer" containerID="596954fad0a196322500865e2449258ae76949fa1d453f28c55e042f149f4205"
Sep 17 18:12:31.722776 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:31.722760    2562 scope.go:117] "RemoveContainer" containerID="91bdc7013705e7f8cf5956941178e4c71e9cba245dbe2d4dac119113e7490d6d"
Sep 17 18:12:31.722906 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:31.722877    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"]
Sep 17 18:12:31.726263 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:31.726238    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-jxckg"]
Sep 17 18:12:33.276604 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.276567    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" path="/var/lib/kubelet/pods/fff8a33b-c9ef-491c-a082-027e9cd21b0b/volumes"
Sep 17 18:12:33.607866 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.607788    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"]
Sep 17 18:12:33.608076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608064    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kserve-container"
Sep 17 18:12:33.608123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608077    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kserve-container"
Sep 17 18:12:33.608123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608090    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="storage-initializer"
Sep 17 18:12:33.608123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608096    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="storage-initializer"
Sep 17 18:12:33.608123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608102    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="storage-initializer"
Sep 17 18:12:33.608123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608108    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="storage-initializer"
Sep 17 18:12:33.608123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608115    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy"
Sep 17 18:12:33.608123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608121    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608130    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kube-rbac-proxy"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608135    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kube-rbac-proxy"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608147    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608152    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608190    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kube-rbac-proxy"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608200    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kube-rbac-proxy"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608207    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff8a33b-c9ef-491c-a082-027e9cd21b0b" containerName="kserve-container"
Sep 17 18:12:33.608338 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.608213    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe896754-7844-4648-95ca-1b94c5902145" containerName="kserve-container"
Sep 17 18:12:33.625407 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.625378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"]
Sep 17 18:12:33.625577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.625504    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.627780 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.627756    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 18:12:33.627941 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.627864    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 18:12:33.729565 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.729532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.729565 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.729565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.729811 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.729597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.729811 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.729657    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.729811 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.729711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wxg\" (UniqueName: \"kubernetes.io/projected/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-api-access-79wxg\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830111 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830072    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wxg\" (UniqueName: \"kubernetes.io/projected/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-api-access-79wxg\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830294 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830294 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830294 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830294 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830490 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:12:33.830359    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-predictor-serving-cert: secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 18:12:33.830490 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:12:33.830423    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls podName:3ba5f86f-1124-4f3f-8c96-b467b7d086b3 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:34.330400916 +0000 UTC m=+2245.517836756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls") pod "isvc-xgboost-predictor-7566d65bf7-jkfzh" (UID: "3ba5f86f-1124-4f3f-8c96-b467b7d086b3") : secret "isvc-xgboost-predictor-serving-cert" not found
Sep 17 18:12:33.830666 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830935 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830901    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.830979 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.830954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:33.838793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:33.838771    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wxg\" (UniqueName: \"kubernetes.io/projected/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-api-access-79wxg\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:34.334547 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:34.334505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:34.337057 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:34.337033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-jkfzh\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:34.536012 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:34.535970    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:34.854310 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:34.854193    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"]
Sep 17 18:12:34.856826 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:12:34.856796    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba5f86f_1124_4f3f_8c96_b467b7d086b3.slice/crio-5940aee6c9dc0e85dd044a3eec6ec9ec29ee6b07b4a870cf437b198933d57af1 WatchSource:0}: Error finding container 5940aee6c9dc0e85dd044a3eec6ec9ec29ee6b07b4a870cf437b198933d57af1: Status 404 returned error can't find the container with id 5940aee6c9dc0e85dd044a3eec6ec9ec29ee6b07b4a870cf437b198933d57af1
Sep 17 18:12:35.726243 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:35.726208    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerStarted","Data":"2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3"}
Sep 17 18:12:35.726726 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:35.726279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerStarted","Data":"5940aee6c9dc0e85dd044a3eec6ec9ec29ee6b07b4a870cf437b198933d57af1"}
Sep 17 18:12:37.671906 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:37.671865    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:12:39.741506 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:39.741473    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerID="2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3" exitCode=0
Sep 17 18:12:39.741901 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:39.741512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerDied","Data":"2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3"}
Sep 17 18:12:47.671776 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:47.671724    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:12:57.672037 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:57.671995    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:12:59.825488 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:59.825452    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerStarted","Data":"8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55"}
Sep 17 18:12:59.825998 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:59.825535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerStarted","Data":"5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb"}
Sep 17 18:12:59.825998 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:59.825793    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:59.825998 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:59.825849    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:12:59.827342 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:59.827310    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:12:59.849578 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:12:59.849500    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podStartSLOduration=3.385403703 podStartE2EDuration="26.849479894s" totalImagesPullingTime="19.523130954s" totalInitContainerRuntime="3.940945237s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:59.848167653 +0000 UTC m=+2271.035603504" watchObservedRunningTime="2026-09-17 18:12:59.849479894 +0000 UTC m=+2271.036915745"
Sep 17 18:13:00.829403 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:00.829364    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:13:05.834674 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:05.834645    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:13:05.835273 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:05.835245    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:13:07.671863 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:07.671828    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:13:15.835475 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:15.835430    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:13:17.671927 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:17.671881    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:13:25.835930 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:25.835876    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:13:27.672054 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:27.672016    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:13:35.835848 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:35.835806    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:13:37.672655 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:37.672623    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:13:45.835928 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:45.835869    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:13:55.835385 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:13:55.835348    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:14:05.835652 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:05.835569    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:14:15.836073 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:15.836042    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:14:24.121009 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:24.120970    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"]
Sep 17 18:14:24.121471 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:24.121223    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" containerID="cri-o://5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb" gracePeriod=30
Sep 17 18:14:24.121471 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:24.121259    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kube-rbac-proxy" containerID="cri-o://8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55" gracePeriod=30
Sep 17 18:14:25.139981 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:25.139948    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerID="8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55" exitCode=2
Sep 17 18:14:25.140352 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:25.140023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerDied","Data":"8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55"}
Sep 17 18:14:25.829964 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:25.829928    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 18:14:25.835293 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:25.835266    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 18:14:28.365368 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.365341    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:14:28.412062 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.411992    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-cabundle-cert\") pod \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") "
Sep 17 18:14:28.412207 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.412060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-rbac-proxy-sar-config\") pod \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") "
Sep 17 18:14:28.412207 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.412091    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79wxg\" (UniqueName: \"kubernetes.io/projected/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-api-access-79wxg\") pod \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") "
Sep 17 18:14:28.412207 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.412125    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls\") pod \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") "
Sep 17 18:14:28.412207 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.412146    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kserve-provision-location\") pod \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\" (UID: \"3ba5f86f-1124-4f3f-8c96-b467b7d086b3\") "
Sep 17 18:14:28.412511 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.412475    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ba5f86f-1124-4f3f-8c96-b467b7d086b3" (UID: "3ba5f86f-1124-4f3f-8c96-b467b7d086b3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:28.412511 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.412493    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ba5f86f-1124-4f3f-8c96-b467b7d086b3" (UID: "3ba5f86f-1124-4f3f-8c96-b467b7d086b3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:14:28.412511 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.412497    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ba5f86f-1124-4f3f-8c96-b467b7d086b3" (UID: "3ba5f86f-1124-4f3f-8c96-b467b7d086b3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:14:28.414233 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.414209    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ba5f86f-1124-4f3f-8c96-b467b7d086b3" (UID: "3ba5f86f-1124-4f3f-8c96-b467b7d086b3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:14:28.414305 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.414285    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-api-access-79wxg" (OuterVolumeSpecName: "kube-api-access-79wxg") pod "3ba5f86f-1124-4f3f-8c96-b467b7d086b3" (UID: "3ba5f86f-1124-4f3f-8c96-b467b7d086b3"). InnerVolumeSpecName "kube-api-access-79wxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:14:28.512555 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.512521    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wxg\" (UniqueName: \"kubernetes.io/projected/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-api-access-79wxg\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:14:28.512555 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.512550    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:14:28.512555 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.512563    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:14:28.512795 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.512575    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:14:28.512795 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:28.512589    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba5f86f-1124-4f3f-8c96-b467b7d086b3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:14:29.161403 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.161368    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerID="5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb" exitCode=0
Sep 17 18:14:29.161550 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.161454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"
Sep 17 18:14:29.161550 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.161447    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerDied","Data":"5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb"}
Sep 17 18:14:29.161634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.161604    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh" event={"ID":"3ba5f86f-1124-4f3f-8c96-b467b7d086b3","Type":"ContainerDied","Data":"5940aee6c9dc0e85dd044a3eec6ec9ec29ee6b07b4a870cf437b198933d57af1"}
Sep 17 18:14:29.161674 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.161635    2562 scope.go:117] "RemoveContainer" containerID="8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55"
Sep 17 18:14:29.173122 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.173104    2562 scope.go:117] "RemoveContainer" containerID="5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb"
Sep 17 18:14:29.181076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.181058    2562 scope.go:117] "RemoveContainer" containerID="2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3"
Sep 17 18:14:29.188408 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.188383    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"]
Sep 17 18:14:29.189318 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.189300    2562 scope.go:117] "RemoveContainer" containerID="8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55"
Sep 17 18:14:29.189610 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:14:29.189588    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55\": container with ID starting with 8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55 not found: ID does not exist" containerID="8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55"
Sep 17 18:14:29.189707 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.189617    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55"} err="failed to get container status \"8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55\": rpc error: code = NotFound desc = could not find container \"8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55\": container with ID starting with 8e91d68176a1f49de980ab8012b2f329825f323ea907607146e73f46a754cd55 not found: ID does not exist"
Sep 17 18:14:29.189707 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.189633    2562 scope.go:117] "RemoveContainer" containerID="5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb"
Sep 17 18:14:29.189886 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:14:29.189868    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb\": container with ID starting with 5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb not found: ID does not exist" containerID="5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb"
Sep 17 18:14:29.189960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.189891    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb"} err="failed to get container status \"5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb\": rpc error: code = NotFound desc = could not find container \"5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb\": container with ID starting with 5508a2c4b48a8015304a4ff1151730bf40385009e624278239ba24422ed82cbb not found: ID does not exist"
Sep 17 18:14:29.189960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.189906    2562 scope.go:117] "RemoveContainer" containerID="2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3"
Sep 17 18:14:29.190184 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:14:29.190167    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3\": container with ID starting with 2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3 not found: ID does not exist" containerID="2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3"
Sep 17 18:14:29.190227 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.190190    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3"} err="failed to get container status \"2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3\": rpc error: code = NotFound desc = could not find container \"2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3\": container with ID starting with 2016edf8b7a4ce044fcb53a372003a32dd438723bcda7dde94207de172c901e3 not found: ID does not exist"
Sep 17 18:14:29.196119 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.196097    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-jkfzh"]
Sep 17 18:14:29.276766 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:14:29.276732    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" path="/var/lib/kubelet/pods/3ba5f86f-1124-4f3f-8c96-b467b7d086b3/volumes"
Sep 17 18:15:14.426647 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:14.426613    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"]
Sep 17 18:15:14.427099 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:14.426855    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" containerID="cri-o://db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4" gracePeriod=30
Sep 17 18:15:14.427099 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:14.426899    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kube-rbac-proxy" containerID="cri-o://7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08" gracePeriod=30
Sep 17 18:15:15.324495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:15.324459    2562 generic.go:356] "Generic (PLEG): container finished" podID="869453aa-6892-499d-8b34-756eb766a718" containerID="7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08" exitCode=2
Sep 17 18:15:15.324679 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:15.324515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerDied","Data":"7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08"}
Sep 17 18:15:17.667124 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:17.667075    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 18:15:17.671407 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:17.671378    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 17 18:15:19.274444 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.274417    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:15:19.340861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.340771    2562 generic.go:356] "Generic (PLEG): container finished" podID="869453aa-6892-499d-8b34-756eb766a718" containerID="db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4" exitCode=0
Sep 17 18:15:19.340861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.340813    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerDied","Data":"db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4"}
Sep 17 18:15:19.340861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.340843    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"
Sep 17 18:15:19.340861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.340863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd" event={"ID":"869453aa-6892-499d-8b34-756eb766a718","Type":"ContainerDied","Data":"fdb410e020e045943d498e47d88a9e60379963366d7c365f47fd41dfb99e7721"}
Sep 17 18:15:19.341195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.340894    2562 scope.go:117] "RemoveContainer" containerID="7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08"
Sep 17 18:15:19.349235 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.349215    2562 scope.go:117] "RemoveContainer" containerID="db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4"
Sep 17 18:15:19.356480 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.356462    2562 scope.go:117] "RemoveContainer" containerID="82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56"
Sep 17 18:15:19.363536 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.363514    2562 scope.go:117] "RemoveContainer" containerID="7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08"
Sep 17 18:15:19.363797 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:15:19.363777    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08\": container with ID starting with 7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08 not found: ID does not exist" containerID="7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08"
Sep 17 18:15:19.363862 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.363805    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08"} err="failed to get container status \"7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08\": rpc error: code = NotFound desc = could not find container \"7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08\": container with ID starting with 7709086e3a6055996038c226b537ccb4559a383f338479a27c2b36f2e48a5c08 not found: ID does not exist"
Sep 17 18:15:19.363862 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.363824    2562 scope.go:117] "RemoveContainer" containerID="db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4"
Sep 17 18:15:19.364132 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:15:19.364107    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4\": container with ID starting with db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4 not found: ID does not exist" containerID="db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4"
Sep 17 18:15:19.364179 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.364139    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4"} err="failed to get container status \"db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4\": rpc error: code = NotFound desc = could not find container \"db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4\": container with ID starting with db7cc8e7494ae311a90b1a1033c60a4741b3ce2ea03cd6786425c6f802a69aa4 not found: ID does not exist"
Sep 17 18:15:19.364179 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.364155    2562 scope.go:117] "RemoveContainer" containerID="82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56"
Sep 17 18:15:19.364375 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:15:19.364358    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56\": container with ID starting with 82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56 not found: ID does not exist" containerID="82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56"
Sep 17 18:15:19.364416 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.364378    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56"} err="failed to get container status \"82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56\": rpc error: code = NotFound desc = could not find container \"82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56\": container with ID starting with 82c2103667d0f4533f791c534fb715cfe7eb45ae9aa7476b13312eb8299c2e56 not found: ID does not exist"
Sep 17 18:15:19.388550 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/869453aa-6892-499d-8b34-756eb766a718-proxy-tls\") pod \"869453aa-6892-499d-8b34-756eb766a718\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") "
Sep 17 18:15:19.388663 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388576    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-kube-rbac-proxy-sar-config\") pod \"869453aa-6892-499d-8b34-756eb766a718\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") "
Sep 17 18:15:19.388663 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388606    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-cabundle-cert\") pod \"869453aa-6892-499d-8b34-756eb766a718\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") "
Sep 17 18:15:19.388663 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388632    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/869453aa-6892-499d-8b34-756eb766a718-kserve-provision-location\") pod \"869453aa-6892-499d-8b34-756eb766a718\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") "
Sep 17 18:15:19.388771 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388672    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5hw5\" (UniqueName: \"kubernetes.io/projected/869453aa-6892-499d-8b34-756eb766a718-kube-api-access-d5hw5\") pod \"869453aa-6892-499d-8b34-756eb766a718\" (UID: \"869453aa-6892-499d-8b34-756eb766a718\") "
Sep 17 18:15:19.389009 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388984    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869453aa-6892-499d-8b34-756eb766a718-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "869453aa-6892-499d-8b34-756eb766a718" (UID: "869453aa-6892-499d-8b34-756eb766a718"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:19.389009 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "869453aa-6892-499d-8b34-756eb766a718" (UID: "869453aa-6892-499d-8b34-756eb766a718"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:19.389009 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.388998    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "869453aa-6892-499d-8b34-756eb766a718" (UID: "869453aa-6892-499d-8b34-756eb766a718"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:19.390569 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.390551    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869453aa-6892-499d-8b34-756eb766a718-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "869453aa-6892-499d-8b34-756eb766a718" (UID: "869453aa-6892-499d-8b34-756eb766a718"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:19.390697 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.390674    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869453aa-6892-499d-8b34-756eb766a718-kube-api-access-d5hw5" (OuterVolumeSpecName: "kube-api-access-d5hw5") pod "869453aa-6892-499d-8b34-756eb766a718" (UID: "869453aa-6892-499d-8b34-756eb766a718"). InnerVolumeSpecName "kube-api-access-d5hw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:19.489190 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.489154    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.489190 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.489183    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/869453aa-6892-499d-8b34-756eb766a718-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.489190 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.489193    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/869453aa-6892-499d-8b34-756eb766a718-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.489416 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.489202    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5hw5\" (UniqueName: \"kubernetes.io/projected/869453aa-6892-499d-8b34-756eb766a718-kube-api-access-d5hw5\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.489416 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.489211    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/869453aa-6892-499d-8b34-756eb766a718-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:15:19.605799 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.605720    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"]
Sep 17 18:15:19.606031 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606017    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="storage-initializer"
Sep 17 18:15:19.606082 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606033    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="storage-initializer"
Sep 17 18:15:19.606082 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606045    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kube-rbac-proxy"
Sep 17 18:15:19.606082 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606052    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kube-rbac-proxy"
Sep 17 18:15:19.606082 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606059    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kube-rbac-proxy"
Sep 17 18:15:19.606082 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606064    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kube-rbac-proxy"
Sep 17 18:15:19.606082 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606079    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="storage-initializer"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606087    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="storage-initializer"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606096    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606102    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606110    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606115    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606152    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kserve-container"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606159    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba5f86f-1124-4f3f-8c96-b467b7d086b3" containerName="kube-rbac-proxy"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606171    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kserve-container"
Sep 17 18:15:19.606302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.606180    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="869453aa-6892-499d-8b34-756eb766a718" containerName="kube-rbac-proxy"
Sep 17 18:15:19.610534 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.610516    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.612641 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.612615    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 18:15:19.612752 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.612696    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:15:19.622010 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.621977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"]
Sep 17 18:15:19.649978 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.649953    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"]
Sep 17 18:15:19.653149 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.653131    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.655065 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.655046    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:15:19.664632 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.664602    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"]
Sep 17 18:15:19.665758 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.665722    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"]
Sep 17 18:15:19.669627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.669606    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-786544c6c8-ghxsd"]
Sep 17 18:15:19.791085 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791014    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkj8\" (UniqueName: \"kubernetes.io/projected/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-api-access-hdkj8\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.791085 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791053    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.791085 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.791381 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zkrx\" (UniqueName: \"kubernetes.io/projected/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-api-access-8zkrx\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.791381 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791147    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.791381 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.791381 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.791381 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.791381 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.791586 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.791438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cb8e65-dbd4-401a-afb2-2cce1af228da-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.891808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.891729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkj8\" (UniqueName: \"kubernetes.io/projected/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-api-access-hdkj8\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.891808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.891766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.891808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.891787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.891808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.891811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zkrx\" (UniqueName: \"kubernetes.io/projected/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-api-access-8zkrx\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.892103 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:15:19.891895    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:15:19.892103 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.891943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.892103 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:15:19.891977    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls podName:49310a5d-63d9-4b1f-b114-0d98f7a7bfe2 nodeName:}" failed. No retries permitted until 2026-09-17 18:15:20.391962454 +0000 UTC m=+2411.579398280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls") pod "isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" (UID: "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:15:19.892103 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.892103 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892061    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.892348 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.892348 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.892348 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.892348 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cb8e65-dbd4-401a-afb2-2cce1af228da-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.892561 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kserve-provision-location\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.892561 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892538    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.892701 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.892844 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-cabundle-cert\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.892942 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.892903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.894583 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.894568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cb8e65-dbd4-401a-afb2-2cce1af228da-proxy-tls\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.900019 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.899999    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkj8\" (UniqueName: \"kubernetes.io/projected/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-api-access-hdkj8\") pod \"isvc-canary-stop-predictor-684df5466b-d8zdt\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:19.900116 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.900029    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zkrx\" (UniqueName: \"kubernetes.io/projected/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-api-access-8zkrx\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:19.962086 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:19.962046    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:20.237330 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:20.237294    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"]
Sep 17 18:15:20.240680 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:15:20.240642    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cb8e65_dbd4_401a_afb2_2cce1af228da.slice/crio-c1335ec494344c3f4d9054d5fd91ef249b23b97418510bfd5004e525d6ccf134 WatchSource:0}: Error finding container c1335ec494344c3f4d9054d5fd91ef249b23b97418510bfd5004e525d6ccf134: Status 404 returned error can't find the container with id c1335ec494344c3f4d9054d5fd91ef249b23b97418510bfd5004e525d6ccf134
Sep 17 18:15:20.347118 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:20.347075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerStarted","Data":"1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6"}
Sep 17 18:15:20.347563 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:20.347145    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerStarted","Data":"c1335ec494344c3f4d9054d5fd91ef249b23b97418510bfd5004e525d6ccf134"}
Sep 17 18:15:20.395713 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:20.395673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:20.398307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:20.398278    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:20.521406 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:20.521324    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:20.793473 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:20.793399    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"]
Sep 17 18:15:20.796527 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:15:20.796498    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49310a5d_63d9_4b1f_b114_0d98f7a7bfe2.slice/crio-937ae8fe7a86d67543eb242fa8d660a206653488a12af42b789b2b908b1ce41f WatchSource:0}: Error finding container 937ae8fe7a86d67543eb242fa8d660a206653488a12af42b789b2b908b1ce41f: Status 404 returned error can't find the container with id 937ae8fe7a86d67543eb242fa8d660a206653488a12af42b789b2b908b1ce41f
Sep 17 18:15:21.275932 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:21.275879    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="869453aa-6892-499d-8b34-756eb766a718" path="/var/lib/kubelet/pods/869453aa-6892-499d-8b34-756eb766a718/volumes"
Sep 17 18:15:21.351988 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:21.351952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerStarted","Data":"6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3"}
Sep 17 18:15:21.352347 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:21.352027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerStarted","Data":"937ae8fe7a86d67543eb242fa8d660a206653488a12af42b789b2b908b1ce41f"}
Sep 17 18:15:24.369654 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:24.369618    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerID="1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6" exitCode=0
Sep 17 18:15:24.370109 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:24.369704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerDied","Data":"1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6"}
Sep 17 18:15:25.374904 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.374822    2562 generic.go:356] "Generic (PLEG): container finished" podID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerID="6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3" exitCode=0
Sep 17 18:15:25.374904 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.374896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerDied","Data":"6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3"}
Sep 17 18:15:25.377254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.377234    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerStarted","Data":"21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4"}
Sep 17 18:15:25.377354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.377282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerStarted","Data":"6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a"}
Sep 17 18:15:25.377550 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.377530    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:25.377603 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.377592    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:25.379031 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.379007    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:15:25.414969 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:25.414885    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podStartSLOduration=2.752386523 podStartE2EDuration="6.414863773s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.66247725s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:25.413417634 +0000 UTC m=+2416.600853514" watchObservedRunningTime="2026-09-17 18:15:25.414863773 +0000 UTC m=+2416.602299622"
Sep 17 18:15:26.384089 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:26.384053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerStarted","Data":"8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119"}
Sep 17 18:15:26.384512 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:26.384119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerStarted","Data":"07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292"}
Sep 17 18:15:26.384512 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:26.384350    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:15:26.384606 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:26.384574    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:26.384713 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:26.384695    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:26.386078 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:26.386055    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:15:26.403055 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:26.403011    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podStartSLOduration=3.218704801 podStartE2EDuration="7.40299743s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.184292629s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:26.401853122 +0000 UTC m=+2417.589288972" watchObservedRunningTime="2026-09-17 18:15:26.40299743 +0000 UTC m=+2417.590433339"
Sep 17 18:15:27.388782 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:27.388743    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:15:31.388732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:31.388659    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:15:31.389310 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:31.389286    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:15:32.393907 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:32.393879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:15:32.394441 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:32.394417    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:15:41.389186 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:41.389145    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:15:42.394894 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:42.394856    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:15:51.390219 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:51.390172    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:15:52.394712 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:15:52.394675    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:16:01.389518 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:01.389475    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:16:02.395135 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:02.395093    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:16:11.390039 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:11.389988    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:16:12.395062 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:12.395026    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:16:19.780624 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.780590    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"]
Sep 17 18:16:19.784160 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.784140    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.786489 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.786466    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:16:19.786589 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.786577    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 18:16:19.796127 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.796106    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"]
Sep 17 18:16:19.849577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.849539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.849753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.849597    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.849807 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.849755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmmk\" (UniqueName: \"kubernetes.io/projected/98f7ab33-46e2-4137-849f-8be74e6defce-kube-api-access-rpmmk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.849807 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.849798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f7ab33-46e2-4137-849f-8be74e6defce-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.849901 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.849856    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f7ab33-46e2-4137-849f-8be74e6defce-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.950470 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.950434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmmk\" (UniqueName: \"kubernetes.io/projected/98f7ab33-46e2-4137-849f-8be74e6defce-kube-api-access-rpmmk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.950667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.950478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f7ab33-46e2-4137-849f-8be74e6defce-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.950667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.950509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f7ab33-46e2-4137-849f-8be74e6defce-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.950667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.950540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.950667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.950567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.951074 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.951044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f7ab33-46e2-4137-849f-8be74e6defce-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.951364 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.951345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.951436 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.951345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.953076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.953055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f7ab33-46e2-4137-849f-8be74e6defce-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:19.959437 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:19.959414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmmk\" (UniqueName: \"kubernetes.io/projected/98f7ab33-46e2-4137-849f-8be74e6defce-kube-api-access-rpmmk\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-r22bv\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:20.094797 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:20.094697    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:20.369309 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:20.369216    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"]
Sep 17 18:16:20.372044 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:16:20.372014    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f7ab33_46e2_4137_849f_8be74e6defce.slice/crio-09da529bfb3a8b62fb7ce3bba03b005443915460d29a348dffaf85d0a7076e6b WatchSource:0}: Error finding container 09da529bfb3a8b62fb7ce3bba03b005443915460d29a348dffaf85d0a7076e6b: Status 404 returned error can't find the container with id 09da529bfb3a8b62fb7ce3bba03b005443915460d29a348dffaf85d0a7076e6b
Sep 17 18:16:20.589711 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:20.589664    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerStarted","Data":"f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2"}
Sep 17 18:16:20.589875 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:20.589745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerStarted","Data":"09da529bfb3a8b62fb7ce3bba03b005443915460d29a348dffaf85d0a7076e6b"}
Sep 17 18:16:21.389611 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:21.389573    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:16:22.394695 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:22.394653    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:16:24.605055 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:24.605027    2562 generic.go:356] "Generic (PLEG): container finished" podID="98f7ab33-46e2-4137-849f-8be74e6defce" containerID="f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2" exitCode=0
Sep 17 18:16:24.605443 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:24.605107    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerDied","Data":"f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2"}
Sep 17 18:16:25.613373 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:25.613337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerStarted","Data":"84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011"}
Sep 17 18:16:25.613852 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:25.613410    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerStarted","Data":"35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16"}
Sep 17 18:16:25.613852 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:25.613630    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:25.637241 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:25.637185    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podStartSLOduration=2.84245 podStartE2EDuration="6.637167124s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794717124s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:25.636592234 +0000 UTC m=+2476.824028084" watchObservedRunningTime="2026-09-17 18:16:25.637167124 +0000 UTC m=+2476.824602973"
Sep 17 18:16:26.617156 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:26.617116    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:26.618368 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:26.618337    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:16:27.621450 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:27.621408    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:16:31.390222 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:31.390184    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:16:32.395144 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:32.395097    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:16:32.625983 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:32.625954    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:16:32.626510 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:32.626481    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:16:37.272278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:37.272235    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 18:16:41.276159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:41.276128    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:16:42.626717 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:42.626678    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:16:47.276462 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:47.276434    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:16:47.395539 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:47.395465    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"]
Sep 17 18:16:47.395688 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:47.395666    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" containerID="cri-o://6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a" gracePeriod=30
Sep 17 18:16:47.395741 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:47.395715    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kube-rbac-proxy" containerID="cri-o://21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4" gracePeriod=30
Sep 17 18:16:47.704193 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:47.704106    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerID="21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4" exitCode=2
Sep 17 18:16:47.704193 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:47.704160    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerDied","Data":"21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4"}
Sep 17 18:16:49.624441 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:49.624409    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"]
Sep 17 18:16:49.624880 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:49.624660    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container" containerID="cri-o://07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292" gracePeriod=30
Sep 17 18:16:49.624880 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:49.624687    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kube-rbac-proxy" containerID="cri-o://8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119" gracePeriod=30
Sep 17 18:16:50.719677 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:50.719645    2562 generic.go:356] "Generic (PLEG): container finished" podID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerID="8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119" exitCode=2
Sep 17 18:16:50.720076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:50.719718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerDied","Data":"8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119"}
Sep 17 18:16:51.272637 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:51.272591    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 18:16:51.385144 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:51.385100    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 18:16:52.330714 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.330687    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:16:52.389189 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.389143    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 17 18:16:52.402822 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.402747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-rbac-proxy-sar-config\") pod \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") "
Sep 17 18:16:52.402822 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.402785    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kserve-provision-location\") pod \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") "
Sep 17 18:16:52.403040 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.402837    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdkj8\" (UniqueName: \"kubernetes.io/projected/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-api-access-hdkj8\") pod \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") "
Sep 17 18:16:52.403040 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.402873    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cb8e65-dbd4-401a-afb2-2cce1af228da-proxy-tls\") pod \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") "
Sep 17 18:16:52.403040 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.402902    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-cabundle-cert\") pod \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\" (UID: \"c3cb8e65-dbd4-401a-afb2-2cce1af228da\") "
Sep 17 18:16:52.403210 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.403184    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3cb8e65-dbd4-401a-afb2-2cce1af228da" (UID: "c3cb8e65-dbd4-401a-afb2-2cce1af228da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:52.403210 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.403197    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3cb8e65-dbd4-401a-afb2-2cce1af228da" (UID: "c3cb8e65-dbd4-401a-afb2-2cce1af228da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:52.403282 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.403201    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3cb8e65-dbd4-401a-afb2-2cce1af228da" (UID: "c3cb8e65-dbd4-401a-afb2-2cce1af228da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:52.405069 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.405038    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-api-access-hdkj8" (OuterVolumeSpecName: "kube-api-access-hdkj8") pod "c3cb8e65-dbd4-401a-afb2-2cce1af228da" (UID: "c3cb8e65-dbd4-401a-afb2-2cce1af228da"). InnerVolumeSpecName "kube-api-access-hdkj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:52.405167 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.405095    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3cb8e65-dbd4-401a-afb2-2cce1af228da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3cb8e65-dbd4-401a-afb2-2cce1af228da" (UID: "c3cb8e65-dbd4-401a-afb2-2cce1af228da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:52.504255 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.504219    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:52.504255 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.504248    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:52.504255 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.504258    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:52.504487 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.504267    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkj8\" (UniqueName: \"kubernetes.io/projected/c3cb8e65-dbd4-401a-afb2-2cce1af228da-kube-api-access-hdkj8\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:52.504487 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.504277    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3cb8e65-dbd4-401a-afb2-2cce1af228da-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:52.626701 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.626667    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:16:52.729896 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.729800    2562 generic.go:356] "Generic (PLEG): container finished" podID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerID="6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a" exitCode=0
Sep 17 18:16:52.729896 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.729860    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"
Sep 17 18:16:52.729896 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.729881    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerDied","Data":"6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a"}
Sep 17 18:16:52.730194 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.729953    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt" event={"ID":"c3cb8e65-dbd4-401a-afb2-2cce1af228da","Type":"ContainerDied","Data":"c1335ec494344c3f4d9054d5fd91ef249b23b97418510bfd5004e525d6ccf134"}
Sep 17 18:16:52.730194 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.729981    2562 scope.go:117] "RemoveContainer" containerID="21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4"
Sep 17 18:16:52.738391 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.738376    2562 scope.go:117] "RemoveContainer" containerID="6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a"
Sep 17 18:16:52.745466 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.745447    2562 scope.go:117] "RemoveContainer" containerID="1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6"
Sep 17 18:16:52.751278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.751253    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"]
Sep 17 18:16:52.755900 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.755788    2562 scope.go:117] "RemoveContainer" containerID="21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4"
Sep 17 18:16:52.757035 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.756983    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-684df5466b-d8zdt"]
Sep 17 18:16:52.757384 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:16:52.757323    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4\": container with ID starting with 21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4 not found: ID does not exist" containerID="21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4"
Sep 17 18:16:52.757384 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.757366    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4"} err="failed to get container status \"21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4\": rpc error: code = NotFound desc = could not find container \"21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4\": container with ID starting with 21c210152abf4e6e8717033fb7a4488b6531832851febdc643a2e2dfe75fb1c4 not found: ID does not exist"
Sep 17 18:16:52.757597 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.757394    2562 scope.go:117] "RemoveContainer" containerID="6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a"
Sep 17 18:16:52.759326 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:16:52.759271    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a\": container with ID starting with 6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a not found: ID does not exist" containerID="6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a"
Sep 17 18:16:52.759326 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.759314    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a"} err="failed to get container status \"6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a\": rpc error: code = NotFound desc = could not find container \"6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a\": container with ID starting with 6e34e043dbeb60a7563ae039973967c63c262b4098c0f4cb5267f9c8cb9d548a not found: ID does not exist"
Sep 17 18:16:52.759515 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.759338    2562 scope.go:117] "RemoveContainer" containerID="1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6"
Sep 17 18:16:52.759673 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:16:52.759650    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6\": container with ID starting with 1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6 not found: ID does not exist" containerID="1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6"
Sep 17 18:16:52.759775 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:52.759691    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6"} err="failed to get container status \"1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6\": rpc error: code = NotFound desc = could not find container \"1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6\": container with ID starting with 1bb14cad27e21db368ce7dceb8fbb6f7a0d3da1428307ef342bbc98539078cb6 not found: ID does not exist"
Sep 17 18:16:53.276700 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:53.276667    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" path="/var/lib/kubelet/pods/c3cb8e65-dbd4-401a-afb2-2cce1af228da/volumes"
Sep 17 18:16:54.569594 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.569572    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:16:54.620667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.620587    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-cabundle-cert\") pod \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") "
Sep 17 18:16:54.620667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.620633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-rbac-proxy-sar-config\") pod \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") "
Sep 17 18:16:54.620667 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.620668    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls\") pod \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") "
Sep 17 18:16:54.620988 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.620699    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kserve-provision-location\") pod \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") "
Sep 17 18:16:54.620988 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.620735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zkrx\" (UniqueName: \"kubernetes.io/projected/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-api-access-8zkrx\") pod \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\" (UID: \"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2\") "
Sep 17 18:16:54.621106 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.620984    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" (UID: "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:54.621106 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.621015    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" (UID: "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:16:54.621106 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.621091    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" (UID: "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:16:54.622820 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.622795    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" (UID: "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:16:54.622899 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.622862    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-api-access-8zkrx" (OuterVolumeSpecName: "kube-api-access-8zkrx") pod "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" (UID: "49310a5d-63d9-4b1f-b114-0d98f7a7bfe2"). InnerVolumeSpecName "kube-api-access-8zkrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:54.721335 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.721300    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zkrx\" (UniqueName: \"kubernetes.io/projected/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-api-access-8zkrx\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.721335 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.721330    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.721335 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.721340    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.721576 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.721350    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.721576 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.721358    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:16:54.741101 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.741074    2562 generic.go:356] "Generic (PLEG): container finished" podID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerID="07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292" exitCode=0
Sep 17 18:16:54.741240 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.741120    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerDied","Data":"07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292"}
Sep 17 18:16:54.741240 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.741133    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"
Sep 17 18:16:54.741240 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.741168    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g" event={"ID":"49310a5d-63d9-4b1f-b114-0d98f7a7bfe2","Type":"ContainerDied","Data":"937ae8fe7a86d67543eb242fa8d660a206653488a12af42b789b2b908b1ce41f"}
Sep 17 18:16:54.741240 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.741196    2562 scope.go:117] "RemoveContainer" containerID="8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119"
Sep 17 18:16:54.749765 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.749742    2562 scope.go:117] "RemoveContainer" containerID="07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292"
Sep 17 18:16:54.756979 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.756959    2562 scope.go:117] "RemoveContainer" containerID="6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3"
Sep 17 18:16:54.764158 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.764140    2562 scope.go:117] "RemoveContainer" containerID="8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119"
Sep 17 18:16:54.764441 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:16:54.764420    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119\": container with ID starting with 8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119 not found: ID does not exist" containerID="8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119"
Sep 17 18:16:54.764509 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.764449    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119"} err="failed to get container status \"8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119\": rpc error: code = NotFound desc = could not find container \"8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119\": container with ID starting with 8910ad35c7894e453634c15394de79df1ede14a8b268410c2ba54eb468513119 not found: ID does not exist"
Sep 17 18:16:54.764509 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.764465    2562 scope.go:117] "RemoveContainer" containerID="07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292"
Sep 17 18:16:54.764694 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:16:54.764680    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292\": container with ID starting with 07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292 not found: ID does not exist" containerID="07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292"
Sep 17 18:16:54.764736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.764698    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292"} err="failed to get container status \"07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292\": rpc error: code = NotFound desc = could not find container \"07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292\": container with ID starting with 07fde463b1e384ccf4f99ee6e00f432b1dfa6c92675596bb9357c1dc8f42f292 not found: ID does not exist"
Sep 17 18:16:54.764736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.764716    2562 scope.go:117] "RemoveContainer" containerID="6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3"
Sep 17 18:16:54.764860 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.764701    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"]
Sep 17 18:16:54.764985 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:16:54.764966    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3\": container with ID starting with 6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3 not found: ID does not exist" containerID="6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3"
Sep 17 18:16:54.765037 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.764991    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3"} err="failed to get container status \"6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3\": rpc error: code = NotFound desc = could not find container \"6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3\": container with ID starting with 6281666ee86a05dfb8b5dca02d15b9ba8d7c8340398c94d0b28338179f332cb3 not found: ID does not exist"
Sep 17 18:16:54.768709 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:54.768689    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-68f75f5f5f-c2m9g"]
Sep 17 18:16:55.276459 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:16:55.276423    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" path="/var/lib/kubelet/pods/49310a5d-63d9-4b1f-b114-0d98f7a7bfe2/volumes"
Sep 17 18:17:02.626755 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:02.626672    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:17:12.626466 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:12.626419    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:17:22.626618 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:22.626577    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:17:32.627095 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:32.627064    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:17:40.295582 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:40.295550    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"]
Sep 17 18:17:40.296035 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:40.295827    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" containerID="cri-o://35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16" gracePeriod=30
Sep 17 18:17:40.296035 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:40.295845    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kube-rbac-proxy" containerID="cri-o://84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011" gracePeriod=30
Sep 17 18:17:40.910677 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:40.910644    2562 generic.go:356] "Generic (PLEG): container finished" podID="98f7ab33-46e2-4137-849f-8be74e6defce" containerID="84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011" exitCode=2
Sep 17 18:17:40.910852 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:40.910702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerDied","Data":"84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011"}
Sep 17 18:17:42.622602 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:42.622563    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 18:17:42.626878 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:42.626856    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 18:17:44.442031 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.442006    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:17:44.597745 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.597635    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f7ab33-46e2-4137-849f-8be74e6defce-proxy-tls\") pod \"98f7ab33-46e2-4137-849f-8be74e6defce\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") "
Sep 17 18:17:44.597745 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.597712    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f7ab33-46e2-4137-849f-8be74e6defce-kserve-provision-location\") pod \"98f7ab33-46e2-4137-849f-8be74e6defce\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") "
Sep 17 18:17:44.597745 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.597742    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-cabundle-cert\") pod \"98f7ab33-46e2-4137-849f-8be74e6defce\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") "
Sep 17 18:17:44.598083 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.597762    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpmmk\" (UniqueName: \"kubernetes.io/projected/98f7ab33-46e2-4137-849f-8be74e6defce-kube-api-access-rpmmk\") pod \"98f7ab33-46e2-4137-849f-8be74e6defce\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") "
Sep 17 18:17:44.598083 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.597781    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-kube-rbac-proxy-sar-config\") pod \"98f7ab33-46e2-4137-849f-8be74e6defce\" (UID: \"98f7ab33-46e2-4137-849f-8be74e6defce\") "
Sep 17 18:17:44.598204 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.598143    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f7ab33-46e2-4137-849f-8be74e6defce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98f7ab33-46e2-4137-849f-8be74e6defce" (UID: "98f7ab33-46e2-4137-849f-8be74e6defce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:44.598204 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.598180    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98f7ab33-46e2-4137-849f-8be74e6defce" (UID: "98f7ab33-46e2-4137-849f-8be74e6defce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:44.598295 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.598208    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98f7ab33-46e2-4137-849f-8be74e6defce" (UID: "98f7ab33-46e2-4137-849f-8be74e6defce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:44.599848 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.599826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f7ab33-46e2-4137-849f-8be74e6defce-kube-api-access-rpmmk" (OuterVolumeSpecName: "kube-api-access-rpmmk") pod "98f7ab33-46e2-4137-849f-8be74e6defce" (UID: "98f7ab33-46e2-4137-849f-8be74e6defce"). InnerVolumeSpecName "kube-api-access-rpmmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:44.599937 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.599906    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f7ab33-46e2-4137-849f-8be74e6defce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98f7ab33-46e2-4137-849f-8be74e6defce" (UID: "98f7ab33-46e2-4137-849f-8be74e6defce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:44.698349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.698315    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f7ab33-46e2-4137-849f-8be74e6defce-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:17:44.698349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.698343    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98f7ab33-46e2-4137-849f-8be74e6defce-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:17:44.698349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.698352    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:17:44.698664 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.698361    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmmk\" (UniqueName: \"kubernetes.io/projected/98f7ab33-46e2-4137-849f-8be74e6defce-kube-api-access-rpmmk\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:17:44.698664 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.698372    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f7ab33-46e2-4137-849f-8be74e6defce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:17:44.926421 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.926337    2562 generic.go:356] "Generic (PLEG): container finished" podID="98f7ab33-46e2-4137-849f-8be74e6defce" containerID="35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16" exitCode=0
Sep 17 18:17:44.926421 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.926398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerDied","Data":"35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16"}
Sep 17 18:17:44.926421 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.926406    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"
Sep 17 18:17:44.926686 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.926456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv" event={"ID":"98f7ab33-46e2-4137-849f-8be74e6defce","Type":"ContainerDied","Data":"09da529bfb3a8b62fb7ce3bba03b005443915460d29a348dffaf85d0a7076e6b"}
Sep 17 18:17:44.926686 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.926484    2562 scope.go:117] "RemoveContainer" containerID="84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011"
Sep 17 18:17:44.939947 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.939904    2562 scope.go:117] "RemoveContainer" containerID="35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16"
Sep 17 18:17:44.946948 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.946932    2562 scope.go:117] "RemoveContainer" containerID="f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2"
Sep 17 18:17:44.951930 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.951891    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"]
Sep 17 18:17:44.954616 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.954599    2562 scope.go:117] "RemoveContainer" containerID="84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011"
Sep 17 18:17:44.954929 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:17:44.954893    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011\": container with ID starting with 84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011 not found: ID does not exist" containerID="84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011"
Sep 17 18:17:44.955004 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.954938    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011"} err="failed to get container status \"84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011\": rpc error: code = NotFound desc = could not find container \"84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011\": container with ID starting with 84ede704c7c8a1ddd85401430ea59cf3e1911a7a6087796a698ad7d9a1438011 not found: ID does not exist"
Sep 17 18:17:44.955004 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.954957    2562 scope.go:117] "RemoveContainer" containerID="35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16"
Sep 17 18:17:44.955234 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:17:44.955205    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16\": container with ID starting with 35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16 not found: ID does not exist" containerID="35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16"
Sep 17 18:17:44.955279 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.955241    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16"} err="failed to get container status \"35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16\": rpc error: code = NotFound desc = could not find container \"35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16\": container with ID starting with 35978ec5e71f08b98080c55c0309498c3f8a03f909d5c0564bd74cef1c100d16 not found: ID does not exist"
Sep 17 18:17:44.955279 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.955255    2562 scope.go:117] "RemoveContainer" containerID="f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2"
Sep 17 18:17:44.955527 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:17:44.955508    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2\": container with ID starting with f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2 not found: ID does not exist" containerID="f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2"
Sep 17 18:17:44.955580 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.955535    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2"} err="failed to get container status \"f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2\": rpc error: code = NotFound desc = could not find container \"f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2\": container with ID starting with f2f792ef397866afe86a8f5e735bbc299d9b642143c127b73f1f27eb516b72a2 not found: ID does not exist"
Sep 17 18:17:44.955619 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:44.955573    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-r22bv"]
Sep 17 18:17:45.276461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:17:45.276429    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" path="/var/lib/kubelet/pods/98f7ab33-46e2-4137-849f-8be74e6defce/volumes"
Sep 17 18:18:50.741948 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.741897    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"]
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742200    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742212    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742221    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742227    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742235    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="storage-initializer"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742241    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="storage-initializer"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742250    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="storage-initializer"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742257    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="storage-initializer"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742266    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="storage-initializer"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742271    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="storage-initializer"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742277    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kube-rbac-proxy"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742282    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kube-rbac-proxy"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742289    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kube-rbac-proxy"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742294    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kube-rbac-proxy"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742301    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kube-rbac-proxy"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742306    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kube-rbac-proxy"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742313    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742318    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742356    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742362    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kube-rbac-proxy"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742368    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f7ab33-46e2-4137-849f-8be74e6defce" containerName="kserve-container"
Sep 17 18:18:50.742360 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742373    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3cb8e65-dbd4-401a-afb2-2cce1af228da" containerName="kube-rbac-proxy"
Sep 17 18:18:50.743064 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742379    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kserve-container"
Sep 17 18:18:50.743064 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.742383    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="49310a5d-63d9-4b1f-b114-0d98f7a7bfe2" containerName="kube-rbac-proxy"
Sep 17 18:18:50.745320 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.745299    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.748008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.747984    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:18:50.748504 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.748482    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:18:50.748624 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.748530    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:50.748624 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.748558    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:50.748624 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.748582    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:18:50.748776 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.748666    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:18:50.748776 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.748705    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 18:18:50.757280 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.757262    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"]
Sep 17 18:18:50.852823 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.852790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0128fcb-4c03-411d-aac4-d02962eb3c39-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.852823 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.852828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0128fcb-4c03-411d-aac4-d02962eb3c39-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.853067 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.852846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.853067 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.852905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcfmj\" (UniqueName: \"kubernetes.io/projected/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-api-access-pcfmj\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.853067 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.852985    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.953589 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.953552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcfmj\" (UniqueName: \"kubernetes.io/projected/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-api-access-pcfmj\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.953789 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.953601    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.953789 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.953633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0128fcb-4c03-411d-aac4-d02962eb3c39-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.953789 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.953651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0128fcb-4c03-411d-aac4-d02962eb3c39-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.953789 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.953666    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.954195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.954173    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0128fcb-4c03-411d-aac4-d02962eb3c39-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.954326 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.954309    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.954388 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.954370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.956395 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.956371    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0128fcb-4c03-411d-aac4-d02962eb3c39-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:50.961688 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:50.961647    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcfmj\" (UniqueName: \"kubernetes.io/projected/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-api-access-pcfmj\") pod \"isvc-xgboost-v2-predictor-fcc668bc-s7ddp\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:51.055749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:51.055657    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:51.328411 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:51.328320    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"]
Sep 17 18:18:51.331135 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:18:51.331099    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0128fcb_4c03_411d_aac4_d02962eb3c39.slice/crio-6a5ca02f6e5d28942e24642d7adfc90c2a2cff568ed5fde3af929b34993bb8af WatchSource:0}: Error finding container 6a5ca02f6e5d28942e24642d7adfc90c2a2cff568ed5fde3af929b34993bb8af: Status 404 returned error can't find the container with id 6a5ca02f6e5d28942e24642d7adfc90c2a2cff568ed5fde3af929b34993bb8af
Sep 17 18:18:52.163213 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:52.163177    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerStarted","Data":"e795f90cdb57356356317584d82c7c409e7b0a07c521b2d247eeefdeda5f8732"}
Sep 17 18:18:52.163610 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:52.163242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerStarted","Data":"6a5ca02f6e5d28942e24642d7adfc90c2a2cff568ed5fde3af929b34993bb8af"}
Sep 17 18:18:56.178067 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:56.178037    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerID="e795f90cdb57356356317584d82c7c409e7b0a07c521b2d247eeefdeda5f8732" exitCode=0
Sep 17 18:18:56.178457 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:56.178074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerDied","Data":"e795f90cdb57356356317584d82c7c409e7b0a07c521b2d247eeefdeda5f8732"}
Sep 17 18:18:57.183883 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:57.183846    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerStarted","Data":"184f08b2718661eca346975017c5a62b0c222a883db14ecccec00108f34b8fa8"}
Sep 17 18:18:57.184328 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:57.183951    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerStarted","Data":"167da25e20d44e0afd3fa09bff88b010d16759c81bee977e079435a783d78bce"}
Sep 17 18:18:57.184328 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:57.184226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:57.184442 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:57.184376    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:18:57.185515 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:57.185488    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:18:57.203243 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:57.203196    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podStartSLOduration=3.335180349 podStartE2EDuration="7.203181131s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.868000782s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:57.201866897 +0000 UTC m=+2628.389302746" watchObservedRunningTime="2026-09-17 18:18:57.203181131 +0000 UTC m=+2628.390616981"
Sep 17 18:18:58.187348 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:58.187306    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:18:59.190253 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:18:59.190204    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:19:00.036508 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.036470    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"]
Sep 17 18:19:00.040056 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.040033    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.042889 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.042866    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 18:19:00.043194 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.043177    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:19:00.051073 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.051050    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"]
Sep 17 18:19:00.123072 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.123036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.123271 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.123108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.123271 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.123142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q669\" (UniqueName: \"kubernetes.io/projected/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-api-access-9q669\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.123271 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.123171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b9141a1-9b70-4db5-a973-0d589dfbefe6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.123271 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.123242    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.223624 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.223580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.224139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.223647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.224139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.223671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q669\" (UniqueName: \"kubernetes.io/projected/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-api-access-9q669\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.224139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.223699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b9141a1-9b70-4db5-a973-0d589dfbefe6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.224139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.223726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.224320 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.224165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.224359 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.224334    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.224394 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.224356    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.226225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.226204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b9141a1-9b70-4db5-a973-0d589dfbefe6-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.234484 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.234465    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q669\" (UniqueName: \"kubernetes.io/projected/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-api-access-9q669\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.350572 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.350481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:00.626375 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:00.626297    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"]
Sep 17 18:19:00.629348 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:19:00.629322    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9141a1_9b70_4db5_a973_0d589dfbefe6.slice/crio-57f7292218be42344245b40cf0b75adb532b5e3ee49a20f5f273e5cbe9717687 WatchSource:0}: Error finding container 57f7292218be42344245b40cf0b75adb532b5e3ee49a20f5f273e5cbe9717687: Status 404 returned error can't find the container with id 57f7292218be42344245b40cf0b75adb532b5e3ee49a20f5f273e5cbe9717687
Sep 17 18:19:01.198737 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:01.198700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerStarted","Data":"5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50"}
Sep 17 18:19:01.198901 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:01.198759    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerStarted","Data":"57f7292218be42344245b40cf0b75adb532b5e3ee49a20f5f273e5cbe9717687"}
Sep 17 18:19:04.194643 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:04.194611    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:19:04.195211 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:04.195183    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:19:05.214869 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:05.214834    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerID="5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50" exitCode=0
Sep 17 18:19:05.215268 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:05.214928    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerDied","Data":"5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50"}
Sep 17 18:19:05.216217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:05.216196    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:19:12.243948 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:12.243898    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerStarted","Data":"c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf"}
Sep 17 18:19:12.244299 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:12.243979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerStarted","Data":"d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14"}
Sep 17 18:19:12.244299 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:12.244170    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:12.268297 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:12.268247    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podStartSLOduration=1.484661106 podStartE2EDuration="12.26822886s" totalImagesPullingTime="6.744998592s" totalInitContainerRuntime="4.038569162s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:12.265999584 +0000 UTC m=+2643.453435433" watchObservedRunningTime="2026-09-17 18:19:12.26822886 +0000 UTC m=+2643.455664709"
Sep 17 18:19:13.248560 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:13.248523    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:13.250008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:13.249977    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:19:14.195800 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:14.195765    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:19:14.252565 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:14.252522    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:19:19.258545 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:19.258514    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:19:19.259195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:19.259167    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:19:24.195636 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:24.195591    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:19:29.259224 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:29.259185    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:19:34.195312 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:34.195272    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:19:39.259441 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:39.259401    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:19:44.195610 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:44.195569    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:19:49.259304 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:49.259263    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:19:54.196066 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:54.196026    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:19:59.259376 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:19:59.259290    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:20:04.195878 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:04.195835    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:20:09.259345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:09.259303    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:20:14.196073 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:14.196043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:20:19.259882 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:19.259836    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:20:21.372860 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:21.372829    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"]
Sep 17 18:20:21.373376 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:21.373081    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" containerID="cri-o://167da25e20d44e0afd3fa09bff88b010d16759c81bee977e079435a783d78bce" gracePeriod=30
Sep 17 18:20:21.373376 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:21.373131    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kube-rbac-proxy" containerID="cri-o://184f08b2718661eca346975017c5a62b0c222a883db14ecccec00108f34b8fa8" gracePeriod=30
Sep 17 18:20:21.505312 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:21.505277    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerID="184f08b2718661eca346975017c5a62b0c222a883db14ecccec00108f34b8fa8" exitCode=2
Sep 17 18:20:21.505457 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:21.505352    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerDied","Data":"184f08b2718661eca346975017c5a62b0c222a883db14ecccec00108f34b8fa8"}
Sep 17 18:20:24.190823 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:24.190779    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 18:20:24.195070 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:24.195048    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 18:20:25.522618 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.522579    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerID="167da25e20d44e0afd3fa09bff88b010d16759c81bee977e079435a783d78bce" exitCode=0
Sep 17 18:20:25.523007 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.522648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerDied","Data":"167da25e20d44e0afd3fa09bff88b010d16759c81bee977e079435a783d78bce"}
Sep 17 18:20:25.610385 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.610361    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:20:25.650054 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650018    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcfmj\" (UniqueName: \"kubernetes.io/projected/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-api-access-pcfmj\") pod \"d0128fcb-4c03-411d-aac4-d02962eb3c39\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") "
Sep 17 18:20:25.650231 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650062    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-rbac-proxy-sar-config\") pod \"d0128fcb-4c03-411d-aac4-d02962eb3c39\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") "
Sep 17 18:20:25.650231 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650102    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-cabundle-cert\") pod \"d0128fcb-4c03-411d-aac4-d02962eb3c39\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") "
Sep 17 18:20:25.650231 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650169    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0128fcb-4c03-411d-aac4-d02962eb3c39-proxy-tls\") pod \"d0128fcb-4c03-411d-aac4-d02962eb3c39\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") "
Sep 17 18:20:25.650231 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650202    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0128fcb-4c03-411d-aac4-d02962eb3c39-kserve-provision-location\") pod \"d0128fcb-4c03-411d-aac4-d02962eb3c39\" (UID: \"d0128fcb-4c03-411d-aac4-d02962eb3c39\") "
Sep 17 18:20:25.650523 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650491    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0128fcb-4c03-411d-aac4-d02962eb3c39" (UID: "d0128fcb-4c03-411d-aac4-d02962eb3c39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:25.650664 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0128fcb-4c03-411d-aac4-d02962eb3c39" (UID: "d0128fcb-4c03-411d-aac4-d02962eb3c39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:25.650664 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.650578    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0128fcb-4c03-411d-aac4-d02962eb3c39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0128fcb-4c03-411d-aac4-d02962eb3c39" (UID: "d0128fcb-4c03-411d-aac4-d02962eb3c39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:25.652238 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.652203    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0128fcb-4c03-411d-aac4-d02962eb3c39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0128fcb-4c03-411d-aac4-d02962eb3c39" (UID: "d0128fcb-4c03-411d-aac4-d02962eb3c39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:25.652646 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.652628    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-api-access-pcfmj" (OuterVolumeSpecName: "kube-api-access-pcfmj") pod "d0128fcb-4c03-411d-aac4-d02962eb3c39" (UID: "d0128fcb-4c03-411d-aac4-d02962eb3c39"). InnerVolumeSpecName "kube-api-access-pcfmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:25.750661 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.750629    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0128fcb-4c03-411d-aac4-d02962eb3c39-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.750661 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.750659    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0128fcb-4c03-411d-aac4-d02962eb3c39-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.750661 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.750668    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcfmj\" (UniqueName: \"kubernetes.io/projected/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-api-access-pcfmj\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.750898 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.750677    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:25.750898 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:25.750686    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0128fcb-4c03-411d-aac4-d02962eb3c39-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:26.530981 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:26.530954    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"
Sep 17 18:20:26.530981 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:26.530950    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp" event={"ID":"d0128fcb-4c03-411d-aac4-d02962eb3c39","Type":"ContainerDied","Data":"6a5ca02f6e5d28942e24642d7adfc90c2a2cff568ed5fde3af929b34993bb8af"}
Sep 17 18:20:26.531517 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:26.531046    2562 scope.go:117] "RemoveContainer" containerID="184f08b2718661eca346975017c5a62b0c222a883db14ecccec00108f34b8fa8"
Sep 17 18:20:26.542458 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:26.542436    2562 scope.go:117] "RemoveContainer" containerID="167da25e20d44e0afd3fa09bff88b010d16759c81bee977e079435a783d78bce"
Sep 17 18:20:26.550349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:26.550328    2562 scope.go:117] "RemoveContainer" containerID="e795f90cdb57356356317584d82c7c409e7b0a07c521b2d247eeefdeda5f8732"
Sep 17 18:20:26.558323 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:26.558299    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"]
Sep 17 18:20:26.564366 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:26.564342    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-s7ddp"]
Sep 17 18:20:27.277113 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:27.277079    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" path="/var/lib/kubelet/pods/d0128fcb-4c03-411d-aac4-d02962eb3c39/volumes"
Sep 17 18:20:29.259902 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:29.259863    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:20:39.259858 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:39.259813    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:20:39.274957 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:39.274920    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 18:20:49.275982 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:49.275954    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:20:51.291815 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:51.291777    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"]
Sep 17 18:20:51.292297 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:51.292090    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kube-rbac-proxy" containerID="cri-o://c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf" gracePeriod=30
Sep 17 18:20:51.292297 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:51.292074    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container" containerID="cri-o://d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14" gracePeriod=30
Sep 17 18:20:51.622214 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:51.622126    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerID="c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf" exitCode=2
Sep 17 18:20:51.622214 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:51.622198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerDied","Data":"c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf"}
Sep 17 18:20:54.253566 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:54.253498    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 18:20:57.125727 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.125701    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:20:57.183599 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.183565    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-cabundle-cert\") pod \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") "
Sep 17 18:20:57.183787 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.183646    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-rbac-proxy-sar-config\") pod \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") "
Sep 17 18:20:57.183787 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.183678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q669\" (UniqueName: \"kubernetes.io/projected/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-api-access-9q669\") pod \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") "
Sep 17 18:20:57.183787 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.183712    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b9141a1-9b70-4db5-a973-0d589dfbefe6-proxy-tls\") pod \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") "
Sep 17 18:20:57.183787 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.183765    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kserve-provision-location\") pod \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\" (UID: \"3b9141a1-9b70-4db5-a973-0d589dfbefe6\") "
Sep 17 18:20:57.184011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.183893    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b9141a1-9b70-4db5-a973-0d589dfbefe6" (UID: "3b9141a1-9b70-4db5-a973-0d589dfbefe6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:57.184154 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.184037    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b9141a1-9b70-4db5-a973-0d589dfbefe6" (UID: "3b9141a1-9b70-4db5-a973-0d589dfbefe6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:20:57.184270 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.184208    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b9141a1-9b70-4db5-a973-0d589dfbefe6" (UID: "3b9141a1-9b70-4db5-a973-0d589dfbefe6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:20:57.185789 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.185768    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9141a1-9b70-4db5-a973-0d589dfbefe6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b9141a1-9b70-4db5-a973-0d589dfbefe6" (UID: "3b9141a1-9b70-4db5-a973-0d589dfbefe6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:20:57.185873 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.185857    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-api-access-9q669" (OuterVolumeSpecName: "kube-api-access-9q669") pod "3b9141a1-9b70-4db5-a973-0d589dfbefe6" (UID: "3b9141a1-9b70-4db5-a973-0d589dfbefe6"). InnerVolumeSpecName "kube-api-access-9q669". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:20:57.284383 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.284358    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:57.284383 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.284384    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:57.284506 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.284393    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:57.284506 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.284402    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q669\" (UniqueName: \"kubernetes.io/projected/3b9141a1-9b70-4db5-a973-0d589dfbefe6-kube-api-access-9q669\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:57.284506 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.284411    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b9141a1-9b70-4db5-a973-0d589dfbefe6-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:20:57.644533 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.644440    2562 generic.go:356] "Generic (PLEG): container finished" podID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerID="d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14" exitCode=0
Sep 17 18:20:57.644533 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.644508    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"
Sep 17 18:20:57.644533 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.644523    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerDied","Data":"d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14"}
Sep 17 18:20:57.644805 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.644586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd" event={"ID":"3b9141a1-9b70-4db5-a973-0d589dfbefe6","Type":"ContainerDied","Data":"57f7292218be42344245b40cf0b75adb532b5e3ee49a20f5f273e5cbe9717687"}
Sep 17 18:20:57.644805 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.644614    2562 scope.go:117] "RemoveContainer" containerID="c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf"
Sep 17 18:20:57.652316 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.652296    2562 scope.go:117] "RemoveContainer" containerID="d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14"
Sep 17 18:20:57.659341 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.659317    2562 scope.go:117] "RemoveContainer" containerID="5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50"
Sep 17 18:20:57.667169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.667145    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"]
Sep 17 18:20:57.667420 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.667406    2562 scope.go:117] "RemoveContainer" containerID="c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf"
Sep 17 18:20:57.667676 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:20:57.667656    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf\": container with ID starting with c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf not found: ID does not exist" containerID="c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf"
Sep 17 18:20:57.667718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.667685    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf"} err="failed to get container status \"c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf\": rpc error: code = NotFound desc = could not find container \"c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf\": container with ID starting with c437fc03ae2cb22c31859496b397a5d7f37b77a91870f1075f98062e7ec5fbbf not found: ID does not exist"
Sep 17 18:20:57.667718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.667703    2562 scope.go:117] "RemoveContainer" containerID="d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14"
Sep 17 18:20:57.667956 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:20:57.667940    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14\": container with ID starting with d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14 not found: ID does not exist" containerID="d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14"
Sep 17 18:20:57.668006 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.667963    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14"} err="failed to get container status \"d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14\": rpc error: code = NotFound desc = could not find container \"d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14\": container with ID starting with d908753645e35b640eb175634f527bd2d71681c3c9b2ce82d4a473860dfc3f14 not found: ID does not exist"
Sep 17 18:20:57.668006 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.667978    2562 scope.go:117] "RemoveContainer" containerID="5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50"
Sep 17 18:20:57.668189 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:20:57.668176    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50\": container with ID starting with 5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50 not found: ID does not exist" containerID="5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50"
Sep 17 18:20:57.668236 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.668193    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50"} err="failed to get container status \"5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50\": rpc error: code = NotFound desc = could not find container \"5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50\": container with ID starting with 5c850b243f4e8b966d05271a28656960ca1ad10e91438e61238b2c781aa34c50 not found: ID does not exist"
Sep 17 18:20:57.671061 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:57.671041    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-w95bd"]
Sep 17 18:20:59.276451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:20:59.276418    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" path="/var/lib/kubelet/pods/3b9141a1-9b70-4db5-a973-0d589dfbefe6/volumes"
Sep 17 18:21:51.715936 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.715886    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"]
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716152    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="storage-initializer"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716169    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="storage-initializer"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716178    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716184    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716201    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716206    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716212    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kube-rbac-proxy"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716217    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kube-rbac-proxy"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716223    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="storage-initializer"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716228    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="storage-initializer"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716235    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kube-rbac-proxy"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716243    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kube-rbac-proxy"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716287    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kserve-container"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716295    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kube-rbac-proxy"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716304    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0128fcb-4c03-411d-aac4-d02962eb3c39" containerName="kube-rbac-proxy"
Sep 17 18:21:51.716349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.716309    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9141a1-9b70-4db5-a973-0d589dfbefe6" containerName="kserve-container"
Sep 17 18:21:51.719269 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.719250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.721877 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.721859    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:21:51.721986 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.721871    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:21:51.721986 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.721897    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:21:51.721986 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.721877    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 18:21:51.721986 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.721961    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:21:51.722797 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.722776    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:21:51.722930 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.722798    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:21:51.730208 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.730189    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"]
Sep 17 18:21:51.879525 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.879486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.879525 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.879525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.879756 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.879554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bjc\" (UniqueName: \"kubernetes.io/projected/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-api-access-57bjc\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.879756 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.879629    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.879756 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.879661    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.980570 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.980484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.980570 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.980520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.980570 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.980550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.980859 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.980630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.980859 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.980703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bjc\" (UniqueName: \"kubernetes.io/projected/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-api-access-57bjc\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.980992 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.980891    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.981261 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.981242    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.981310 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.981282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.983011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.982992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:51.989772 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:51.989748    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bjc\" (UniqueName: \"kubernetes.io/projected/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-api-access-57bjc\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:52.029615 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:52.029582    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:52.301440 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:52.301278    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"]
Sep 17 18:21:52.304209 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:21:52.304184    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60bf33d_e3e0_49cd_93c0_3cc0bbdaddca.slice/crio-199b692830092bd7f3fdda618334fd6a6914f65d26a6531bf79d0426ee6d6f53 WatchSource:0}: Error finding container 199b692830092bd7f3fdda618334fd6a6914f65d26a6531bf79d0426ee6d6f53: Status 404 returned error can't find the container with id 199b692830092bd7f3fdda618334fd6a6914f65d26a6531bf79d0426ee6d6f53
Sep 17 18:21:52.847155 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:52.847115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerStarted","Data":"23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd"}
Sep 17 18:21:52.847562 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:52.847201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerStarted","Data":"199b692830092bd7f3fdda618334fd6a6914f65d26a6531bf79d0426ee6d6f53"}
Sep 17 18:21:56.835991 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.835954    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"]
Sep 17 18:21:56.839236 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.839218    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:56.841705 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.841676    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 18:21:56.841812 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.841692    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:21:56.841865 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.841809    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:21:56.851734 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.851710    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"]
Sep 17 18:21:56.863619 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.863597    2562 generic.go:356] "Generic (PLEG): container finished" podID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerID="23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd" exitCode=0
Sep 17 18:21:56.863729 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.863668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerDied","Data":"23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd"}
Sep 17 18:21:56.915297 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.915271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5lm7\" (UniqueName: \"kubernetes.io/projected/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-api-access-w5lm7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:56.915409 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.915359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:56.915464 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.915433    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:56.915508 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.915494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:56.915580 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:56.915564    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.016726 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.016691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.016992 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.016961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.017100 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.017014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.017100 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.017094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.017215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.017150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5lm7\" (UniqueName: \"kubernetes.io/projected/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-api-access-w5lm7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.017639 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:21:57.017375    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 18:21:57.017639 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:21:57.017448    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls podName:63fb0f81-81ff-453f-9412-fe65b4b43fdd nodeName:}" failed. No retries permitted until 2026-09-17 18:21:57.51742845 +0000 UTC m=+2808.704864287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" (UID: "63fb0f81-81ff-453f-9412-fe65b4b43fdd") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 18:21:57.017895 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.017871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.017996 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.017979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.018170 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.018147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.025699 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.025675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5lm7\" (UniqueName: \"kubernetes.io/projected/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-api-access-w5lm7\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.521101 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.521059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.523579 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.523552    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.749584 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.749548    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:21:57.869343 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.869310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerStarted","Data":"b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0"}
Sep 17 18:21:57.869742 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.869368    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerStarted","Data":"869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5"}
Sep 17 18:21:57.869742 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.869638    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:57.869845 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.869809    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:21:57.870934 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.870891    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:21:57.895400 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:57.895350    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" podStartSLOduration=3.027383297 podStartE2EDuration="6.895334775s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867951478s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:57.893783597 +0000 UTC m=+2809.081219459" watchObservedRunningTime="2026-09-17 18:21:57.895334775 +0000 UTC m=+2809.082770627"
Sep 17 18:21:58.028312 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:58.027952    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"]
Sep 17 18:21:58.030713 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:21:58.030686    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fb0f81_81ff_453f_9412_fe65b4b43fdd.slice/crio-2714db05d916cf23dc8b370273df33eefcbf3b37037326e02b647e1f207751ae WatchSource:0}: Error finding container 2714db05d916cf23dc8b370273df33eefcbf3b37037326e02b647e1f207751ae: Status 404 returned error can't find the container with id 2714db05d916cf23dc8b370273df33eefcbf3b37037326e02b647e1f207751ae
Sep 17 18:21:58.875576 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:58.875531    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerStarted","Data":"03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8"}
Sep 17 18:21:58.876106 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:58.875611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerStarted","Data":"2714db05d916cf23dc8b370273df33eefcbf3b37037326e02b647e1f207751ae"}
Sep 17 18:21:58.876106 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:58.876052    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:21:59.880341 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:59.880308    2562 generic.go:356] "Generic (PLEG): container finished" podID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerID="03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8" exitCode=0
Sep 17 18:21:59.880739 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:21:59.880354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerDied","Data":"03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8"}
Sep 17 18:22:00.887053 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:00.887013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerStarted","Data":"914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba"}
Sep 17 18:22:00.887549 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:00.887078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerStarted","Data":"c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b"}
Sep 17 18:22:00.887549 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:00.887208    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:22:00.887549 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:00.887264    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:22:00.888354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:00.888327    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:22:00.906846 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:00.906796    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podStartSLOduration=3.577032042 podStartE2EDuration="4.906782249s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.329750207s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:00.905364087 +0000 UTC m=+2812.092799937" watchObservedRunningTime="2026-09-17 18:22:00.906782249 +0000 UTC m=+2812.094218155"
Sep 17 18:22:01.890899 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:01.890848    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:22:03.880424 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:03.880389    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:22:03.881024 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:03.880997    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 18:22:06.895501 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:06.895470    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:22:06.896146 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:06.896119    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:22:13.882309 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:13.882279    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:22:16.896769 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:16.896728    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:22:21.820056 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:21.820013    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"]
Sep 17 18:22:21.820570 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:21.820306    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kserve-container" containerID="cri-o://869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5" gracePeriod=30
Sep 17 18:22:21.820570 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:21.820347    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kube-rbac-proxy" containerID="cri-o://b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0" gracePeriod=30
Sep 17 18:22:21.970033 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:21.970002    2562 generic.go:356] "Generic (PLEG): container finished" podID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerID="b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0" exitCode=2
Sep 17 18:22:21.970205 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:21.970075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerDied","Data":"b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0"}
Sep 17 18:22:22.559288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.559263    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:22:22.715737 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.715640    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kserve-provision-location\") pod \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") "
Sep 17 18:22:22.715737 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.715688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-cabundle-cert\") pod \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") "
Sep 17 18:22:22.715737 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.715727    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-proxy-tls\") pod \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") "
Sep 17 18:22:22.716049 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.715757    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57bjc\" (UniqueName: \"kubernetes.io/projected/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-api-access-57bjc\") pod \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") "
Sep 17 18:22:22.716049 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.715829    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-rbac-proxy-sar-config\") pod \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\" (UID: \"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca\") "
Sep 17 18:22:22.716049 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.716023    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" (UID: "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:22.716150 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.716125    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" (UID: "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:22.716225 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.716204    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" (UID: "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:22.717989 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.717954    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-api-access-57bjc" (OuterVolumeSpecName: "kube-api-access-57bjc") pod "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" (UID: "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca"). InnerVolumeSpecName "kube-api-access-57bjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:22.717989 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.717967    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" (UID: "e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:22.816599 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.816565    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.816599 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.816592    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.816599 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.816603    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.816827 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.816613    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.816827 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.816622    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57bjc\" (UniqueName: \"kubernetes.io/projected/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca-kube-api-access-57bjc\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:22:22.975636 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.975548    2562 generic.go:356] "Generic (PLEG): container finished" podID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerID="869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5" exitCode=0
Sep 17 18:22:22.975636 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.975603    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"
Sep 17 18:22:22.976146 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.975639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerDied","Data":"869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5"}
Sep 17 18:22:22.976146 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.975709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4" event={"ID":"e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca","Type":"ContainerDied","Data":"199b692830092bd7f3fdda618334fd6a6914f65d26a6531bf79d0426ee6d6f53"}
Sep 17 18:22:22.976146 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.975751    2562 scope.go:117] "RemoveContainer" containerID="b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0"
Sep 17 18:22:22.984116 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.984098    2562 scope.go:117] "RemoveContainer" containerID="869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5"
Sep 17 18:22:22.990994 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.990934    2562 scope.go:117] "RemoveContainer" containerID="23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd"
Sep 17 18:22:22.996759 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.996735    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"]
Sep 17 18:22:22.998070 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.998056    2562 scope.go:117] "RemoveContainer" containerID="b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0"
Sep 17 18:22:22.998329 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:22:22.998312    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0\": container with ID starting with b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0 not found: ID does not exist" containerID="b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0"
Sep 17 18:22:22.998368 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.998338    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0"} err="failed to get container status \"b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0\": rpc error: code = NotFound desc = could not find container \"b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0\": container with ID starting with b80c46bb0239cdcf54c51ada302ab4f110f366956b8ff6fe9c39241b145f1df0 not found: ID does not exist"
Sep 17 18:22:22.998368 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.998355    2562 scope.go:117] "RemoveContainer" containerID="869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5"
Sep 17 18:22:22.998591 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:22:22.998577    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5\": container with ID starting with 869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5 not found: ID does not exist" containerID="869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5"
Sep 17 18:22:22.998627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.998597    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5"} err="failed to get container status \"869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5\": rpc error: code = NotFound desc = could not find container \"869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5\": container with ID starting with 869735f6c43ea5a69a5b2b5fe3fc05a0c275661eee018b68c7099d59ffdf6eb5 not found: ID does not exist"
Sep 17 18:22:22.998627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.998609    2562 scope.go:117] "RemoveContainer" containerID="23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd"
Sep 17 18:22:22.998814 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:22:22.998799    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd\": container with ID starting with 23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd not found: ID does not exist" containerID="23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd"
Sep 17 18:22:22.998850 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:22.998819    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd"} err="failed to get container status \"23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd\": rpc error: code = NotFound desc = could not find container \"23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd\": container with ID starting with 23d368dd5fd5ec7ef249c6b6df0dcfb6a4383209af8893b7eec6483399dccbdd not found: ID does not exist"
Sep 17 18:22:23.001449 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:23.001428    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-4fvt4"]
Sep 17 18:22:23.276340 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:23.276306    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" path="/var/lib/kubelet/pods/e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca/volumes"
Sep 17 18:22:26.896299 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:26.896263    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:22:36.896277 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:36.896239    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:22:46.896193 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:46.896149    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:22:56.896684 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:22:56.896640    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:23:06.896532 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:06.896451    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:23:16.897074 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:16.897043    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:23:22.299819 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.299781    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"]
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300049    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kserve-container"
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300062    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kserve-container"
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300080    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kube-rbac-proxy"
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300085    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kube-rbac-proxy"
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300095    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="storage-initializer"
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300102    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="storage-initializer"
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300141    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kube-rbac-proxy"
Sep 17 18:23:22.300215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.300148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60bf33d-e3e0-49cd-93c0-3cc0bbdaddca" containerName="kserve-container"
Sep 17 18:23:22.303197 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.303178    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.305595 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.305563    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 17 18:23:22.305730 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.305619    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 17 18:23:22.305786 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.305745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:23:22.318513 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.318482    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"]
Sep 17 18:23:22.458048 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.458009    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.458048 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.458045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wm9m\" (UniqueName: \"kubernetes.io/projected/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-api-access-4wm9m\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.458259 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.458084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba3d394-bf0e-4258-84e8-6e07d74476e5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.458259 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.458106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.458259 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.458129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.559443 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.559371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba3d394-bf0e-4258-84e8-6e07d74476e5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.559443 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.559414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.559565 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.559443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.559565 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.559484    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.559565 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.559508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wm9m\" (UniqueName: \"kubernetes.io/projected/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-api-access-4wm9m\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.559857 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.559834    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.560175 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.560156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.560273 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.560255    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.561840 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.561823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba3d394-bf0e-4258-84e8-6e07d74476e5-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.568363 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.568338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wm9m\" (UniqueName: \"kubernetes.io/projected/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-api-access-4wm9m\") pod \"isvc-sklearn-mcp-predictor-545b85879c-8gtz7\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.617591 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.617568    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:22.890015 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:22.889880    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"]
Sep 17 18:23:22.892684 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:23:22.892655    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba3d394_bf0e_4258_84e8_6e07d74476e5.slice/crio-26a296f874f04ec3013d09210cb7a4b0d7a345059899395d1098b0988c01fd16 WatchSource:0}: Error finding container 26a296f874f04ec3013d09210cb7a4b0d7a345059899395d1098b0988c01fd16: Status 404 returned error can't find the container with id 26a296f874f04ec3013d09210cb7a4b0d7a345059899395d1098b0988c01fd16
Sep 17 18:23:23.190413 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:23.190318    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerStarted","Data":"f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0"}
Sep 17 18:23:23.190413 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:23.190378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerStarted","Data":"26a296f874f04ec3013d09210cb7a4b0d7a345059899395d1098b0988c01fd16"}
Sep 17 18:23:26.856038 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:26.855999    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"]
Sep 17 18:23:26.856512 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:26.856296    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" containerID="cri-o://c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b" gracePeriod=30
Sep 17 18:23:26.856512 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:26.856370    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kube-rbac-proxy" containerID="cri-o://914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba" gracePeriod=30
Sep 17 18:23:26.891698 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:26.891664    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 18:23:26.896021 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:26.895999    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 18:23:27.206465 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:27.206386    2562 generic.go:356] "Generic (PLEG): container finished" podID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerID="914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba" exitCode=2
Sep 17 18:23:27.206615 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:27.206459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerDied","Data":"914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba"}
Sep 17 18:23:27.207864 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:27.207846    2562 generic.go:356] "Generic (PLEG): container finished" podID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerID="f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0" exitCode=0
Sep 17 18:23:27.207986 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:27.207885    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerDied","Data":"f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0"}
Sep 17 18:23:28.215198 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:28.215155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerStarted","Data":"ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29"}
Sep 17 18:23:30.226285 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:30.226257    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerStarted","Data":"94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d"}
Sep 17 18:23:31.232567 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:31.232530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerStarted","Data":"f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23"}
Sep 17 18:23:31.232988 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:31.232635    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:31.232988 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:31.232701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:31.232988 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:31.232743    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:31.255921 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:31.255861    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podStartSLOduration=2.379540315 podStartE2EDuration="9.255840608s" totalImagesPullingTime="2.848180605s" totalInitContainerRuntime="4.028119688s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:31.254116608 +0000 UTC m=+2902.441552486" watchObservedRunningTime="2026-09-17 18:23:31.255840608 +0000 UTC m=+2902.443276463"
Sep 17 18:23:31.907726 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:31.907700    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:23:32.034931 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.034826    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5lm7\" (UniqueName: \"kubernetes.io/projected/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-api-access-w5lm7\") pod \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") "
Sep 17 18:23:32.034931 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.034882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls\") pod \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") "
Sep 17 18:23:32.035230 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.034961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kserve-provision-location\") pod \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") "
Sep 17 18:23:32.035230 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.034993    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-rbac-proxy-sar-config\") pod \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") "
Sep 17 18:23:32.035230 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.035038    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-cabundle-cert\") pod \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\" (UID: \"63fb0f81-81ff-453f-9412-fe65b4b43fdd\") "
Sep 17 18:23:32.035389 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.035349    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63fb0f81-81ff-453f-9412-fe65b4b43fdd" (UID: "63fb0f81-81ff-453f-9412-fe65b4b43fdd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:32.035389 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.035348    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63fb0f81-81ff-453f-9412-fe65b4b43fdd" (UID: "63fb0f81-81ff-453f-9412-fe65b4b43fdd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:32.035489 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.035413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63fb0f81-81ff-453f-9412-fe65b4b43fdd" (UID: "63fb0f81-81ff-453f-9412-fe65b4b43fdd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:32.037078 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.037058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63fb0f81-81ff-453f-9412-fe65b4b43fdd" (UID: "63fb0f81-81ff-453f-9412-fe65b4b43fdd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:32.037177 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.037083    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-api-access-w5lm7" (OuterVolumeSpecName: "kube-api-access-w5lm7") pod "63fb0f81-81ff-453f-9412-fe65b4b43fdd" (UID: "63fb0f81-81ff-453f-9412-fe65b4b43fdd"). InnerVolumeSpecName "kube-api-access-w5lm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:32.136523 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.136493    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63fb0f81-81ff-453f-9412-fe65b4b43fdd-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:23:32.136523 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.136524    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:23:32.136717 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.136538    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:23:32.136717 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.136549    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63fb0f81-81ff-453f-9412-fe65b4b43fdd-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:23:32.136717 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.136562    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5lm7\" (UniqueName: \"kubernetes.io/projected/63fb0f81-81ff-453f-9412-fe65b4b43fdd-kube-api-access-w5lm7\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:23:32.237717 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.237688    2562 generic.go:356] "Generic (PLEG): container finished" podID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerID="c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b" exitCode=0
Sep 17 18:23:32.238086 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.237745    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"
Sep 17 18:23:32.238086 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.237774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerDied","Data":"c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b"}
Sep 17 18:23:32.238086 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.237835    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" event={"ID":"63fb0f81-81ff-453f-9412-fe65b4b43fdd","Type":"ContainerDied","Data":"2714db05d916cf23dc8b370273df33eefcbf3b37037326e02b647e1f207751ae"}
Sep 17 18:23:32.238086 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.237866    2562 scope.go:117] "RemoveContainer" containerID="914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba"
Sep 17 18:23:32.249073 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.249049    2562 scope.go:117] "RemoveContainer" containerID="c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b"
Sep 17 18:23:32.256861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.256842    2562 scope.go:117] "RemoveContainer" containerID="03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8"
Sep 17 18:23:32.263170 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.263148    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"]
Sep 17 18:23:32.264938 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.264891    2562 scope.go:117] "RemoveContainer" containerID="914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba"
Sep 17 18:23:32.265278 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:23:32.265243    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba\": container with ID starting with 914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba not found: ID does not exist" containerID="914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba"
Sep 17 18:23:32.265278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.265284    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba"} err="failed to get container status \"914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba\": rpc error: code = NotFound desc = could not find container \"914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba\": container with ID starting with 914f095da89305373e75c50cf51478666455edf41a8d676a6941ecaf9f3e21ba not found: ID does not exist"
Sep 17 18:23:32.265518 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.265309    2562 scope.go:117] "RemoveContainer" containerID="c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b"
Sep 17 18:23:32.265871 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:23:32.265851    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b\": container with ID starting with c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b not found: ID does not exist" containerID="c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b"
Sep 17 18:23:32.265999 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.265883    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b"} err="failed to get container status \"c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b\": rpc error: code = NotFound desc = could not find container \"c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b\": container with ID starting with c5a5a911e082b533317d2db30b548c654b6a286e6e5aef9bbb55732c88f1641b not found: ID does not exist"
Sep 17 18:23:32.265999 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.265898    2562 scope.go:117] "RemoveContainer" containerID="03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8"
Sep 17 18:23:32.266228 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:23:32.266207    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8\": container with ID starting with 03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8 not found: ID does not exist" containerID="03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8"
Sep 17 18:23:32.266306 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.266231    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8"} err="failed to get container status \"03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8\": rpc error: code = NotFound desc = could not find container \"03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8\": container with ID starting with 03c2314d75c73746b5490bc2477049857c72b54434aa3f7347424019b4df9df8 not found: ID does not exist"
Sep 17 18:23:32.266638 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.266619    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t"]
Sep 17 18:23:32.892579 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:32.892537    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-684ccf8d5d-q6m8t" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": context deadline exceeded"
Sep 17 18:23:33.276241 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:33.276212    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" path="/var/lib/kubelet/pods/63fb0f81-81ff-453f-9412-fe65b4b43fdd/volumes"
Sep 17 18:23:37.244115 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:37.244084    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:23:39.100698 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.100664    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"]
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.100940    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="storage-initializer"
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.100955    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="storage-initializer"
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.100966    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container"
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.100972    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container"
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.100982    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kube-rbac-proxy"
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.100988    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kube-rbac-proxy"
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.101037    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kserve-container"
Sep 17 18:23:39.101159 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.101044    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="63fb0f81-81ff-453f-9412-fe65b4b43fdd" containerName="kube-rbac-proxy"
Sep 17 18:23:39.105147 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.105126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.107605 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.107581    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:23:39.107605 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.107593    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:23:39.107785 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.107593    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 18:23:39.124256 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.124227    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"]
Sep 17 18:23:39.293188 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.293159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.293359 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.293206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.293359 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.293257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppl9h\" (UniqueName: \"kubernetes.io/projected/f6465dc9-4305-4b94-b736-0310068182ba-kube-api-access-ppl9h\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.293359 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.293311    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.293359 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.293331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6465dc9-4305-4b94-b736-0310068182ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394207 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394122    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394207 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppl9h\" (UniqueName: \"kubernetes.io/projected/f6465dc9-4305-4b94-b736-0310068182ba-kube-api-access-ppl9h\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394207 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394469 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394218    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6465dc9-4305-4b94-b736-0310068182ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394469 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394252    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394469 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:23:39.394362    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 18:23:39.394469 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:23:39.394458    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls podName:f6465dc9-4305-4b94-b736-0310068182ba nodeName:}" failed. No retries permitted until 2026-09-17 18:23:39.894436579 +0000 UTC m=+2911.081872428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" (UID: "f6465dc9-4305-4b94-b736-0310068182ba") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 18:23:39.394719 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6465dc9-4305-4b94-b736-0310068182ba-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394828 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.394901 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.394884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.404245 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.404222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppl9h\" (UniqueName: \"kubernetes.io/projected/f6465dc9-4305-4b94-b736-0310068182ba-kube-api-access-ppl9h\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.899560 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.899524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:39.901947 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:39.901907    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:40.015222 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:40.015185    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:40.293902 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:40.293870    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"]
Sep 17 18:23:40.296898 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:23:40.296870    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6465dc9_4305_4b94_b736_0310068182ba.slice/crio-48e73ad3ac7a7f31a79c1501330832b10e4ef9f14cf49888c03886b0e66853fc WatchSource:0}: Error finding container 48e73ad3ac7a7f31a79c1501330832b10e4ef9f14cf49888c03886b0e66853fc: Status 404 returned error can't find the container with id 48e73ad3ac7a7f31a79c1501330832b10e4ef9f14cf49888c03886b0e66853fc
Sep 17 18:23:41.279238 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:41.279206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerStarted","Data":"531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6"}
Sep 17 18:23:41.279409 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:41.279261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerStarted","Data":"48e73ad3ac7a7f31a79c1501330832b10e4ef9f14cf49888c03886b0e66853fc"}
Sep 17 18:23:42.284670 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:42.284639    2562 generic.go:356] "Generic (PLEG): container finished" podID="f6465dc9-4305-4b94-b736-0310068182ba" containerID="531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6" exitCode=0
Sep 17 18:23:42.285088 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:42.284704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerDied","Data":"531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6"}
Sep 17 18:23:43.289508 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:43.289478    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerStarted","Data":"4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352"}
Sep 17 18:23:43.290013 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:43.289524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerStarted","Data":"10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4"}
Sep 17 18:23:43.290013 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:43.289617    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:43.309456 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:43.309412    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podStartSLOduration=3.238677632 podStartE2EDuration="4.309398062s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.07072043s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:43.308165315 +0000 UTC m=+2914.495601164" watchObservedRunningTime="2026-09-17 18:23:43.309398062 +0000 UTC m=+2914.496833910"
Sep 17 18:23:44.292830 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:44.292790    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:44.294071 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:44.294043    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:23:45.295836 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:45.295793    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:23:50.300931 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:50.300879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:23:50.301502 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:50.301407    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:23:57.245863 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:23:57.245823    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:24:00.302399 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:00.302355    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:24:02.618714 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:02.618665    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:24:07.248557 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:07.248520    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:24:10.301625 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:10.301593    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:24:20.301655 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:20.301616    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:24:30.301520 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:30.301445    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:24:37.250349 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:37.250323    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:24:40.302202 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:40.302163    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:24:42.334389 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:42.334357    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"]
Sep 17 18:24:42.334833 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:42.334609    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container" containerID="cri-o://ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29" gracePeriod=30
Sep 17 18:24:42.334833 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:42.334642    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy" containerID="cri-o://f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23" gracePeriod=30
Sep 17 18:24:42.334833 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:42.334667    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-agent" containerID="cri-o://94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d" gracePeriod=30
Sep 17 18:24:42.505436 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:42.505406    2562 generic.go:356] "Generic (PLEG): container finished" podID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerID="f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23" exitCode=2
Sep 17 18:24:42.505575 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:42.505460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerDied","Data":"f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23"}
Sep 17 18:24:45.518386 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:45.518352    2562 generic.go:356] "Generic (PLEG): container finished" podID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerID="ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29" exitCode=0
Sep 17 18:24:45.518748 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:45.518424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerDied","Data":"ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29"}
Sep 17 18:24:47.238772 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:47.238729    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:24:47.244855 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:47.244824    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:24:50.302337 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:50.302301    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:24:52.238580 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:52.238539    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:24:57.239128 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:57.239090    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:24:57.239537 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:57.239224    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:24:57.244952 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:24:57.244908    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:25:00.302140 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:00.302107    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:25:02.239120 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:02.239071    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:25:07.239292 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:07.239253    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:25:07.244746 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:07.244713    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 18:25:07.245271 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:07.245232    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:25:09.121495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:09.121464    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"]
Sep 17 18:25:09.121896 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:09.121694    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" containerID="cri-o://10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4" gracePeriod=30
Sep 17 18:25:09.121896 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:09.121796    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kube-rbac-proxy" containerID="cri-o://4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352" gracePeriod=30
Sep 17 18:25:09.606495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:09.606459    2562 generic.go:356] "Generic (PLEG): container finished" podID="f6465dc9-4305-4b94-b736-0310068182ba" containerID="4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352" exitCode=2
Sep 17 18:25:09.606679 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:09.606536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerDied","Data":"4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352"}
Sep 17 18:25:10.251342 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.251307    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"]
Sep 17 18:25:10.254673 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.254652    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.257414 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.257392    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 18:25:10.257531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.257432    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:25:10.265837 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.265814    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"]
Sep 17 18:25:10.296337 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.296305    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 18:25:10.301687 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.301660    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 18:25:10.364891 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.364860    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tll45\" (UniqueName: \"kubernetes.io/projected/531b7934-4fc4-4167-996d-2c20785450f3-kube-api-access-tll45\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.364891 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.364896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/531b7934-4fc4-4167-996d-2c20785450f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.365087 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.364936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.365087 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.364963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/531b7934-4fc4-4167-996d-2c20785450f3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.466184 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.466151    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tll45\" (UniqueName: \"kubernetes.io/projected/531b7934-4fc4-4167-996d-2c20785450f3-kube-api-access-tll45\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.466334 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.466189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/531b7934-4fc4-4167-996d-2c20785450f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.466334 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.466214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.466334 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.466246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/531b7934-4fc4-4167-996d-2c20785450f3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.466485 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:10.466336    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:25:10.466485 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:10.466396    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls podName:531b7934-4fc4-4167-996d-2c20785450f3 nodeName:}" failed. No retries permitted until 2026-09-17 18:25:10.966378464 +0000 UTC m=+3002.153814291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" (UID: "531b7934-4fc4-4167-996d-2c20785450f3") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:25:10.466659 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.466642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/531b7934-4fc4-4167-996d-2c20785450f3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.466847 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.466830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/531b7934-4fc4-4167-996d-2c20785450f3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.475195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.475175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tll45\" (UniqueName: \"kubernetes.io/projected/531b7934-4fc4-4167-996d-2c20785450f3-kube-api-access-tll45\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.970005 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.969968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:10.972505 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:10.972480    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:11.164853 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:11.164803    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:11.437016 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:11.436987    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"]
Sep 17 18:25:11.439675 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:25:11.439648    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531b7934_4fc4_4167_996d_2c20785450f3.slice/crio-0cbf0cd8f6d7ba3abe2eb6dab8d72c54233dbc838d6a5efb0ab37eec8504e65b WatchSource:0}: Error finding container 0cbf0cd8f6d7ba3abe2eb6dab8d72c54233dbc838d6a5efb0ab37eec8504e65b: Status 404 returned error can't find the container with id 0cbf0cd8f6d7ba3abe2eb6dab8d72c54233dbc838d6a5efb0ab37eec8504e65b
Sep 17 18:25:11.616718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:11.616682    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" event={"ID":"531b7934-4fc4-4167-996d-2c20785450f3","Type":"ContainerStarted","Data":"d7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430"}
Sep 17 18:25:11.616906 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:11.616743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" event={"ID":"531b7934-4fc4-4167-996d-2c20785450f3","Type":"ContainerStarted","Data":"0cbf0cd8f6d7ba3abe2eb6dab8d72c54233dbc838d6a5efb0ab37eec8504e65b"}
Sep 17 18:25:12.239570 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.239534    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 18:25:12.515177 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.515153    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:25:12.622391 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.622354    2562 generic.go:356] "Generic (PLEG): container finished" podID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerID="94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d" exitCode=137
Sep 17 18:25:12.622544 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.622412    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"
Sep 17 18:25:12.622544 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.622432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerDied","Data":"94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d"}
Sep 17 18:25:12.622544 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.622493    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7" event={"ID":"6ba3d394-bf0e-4258-84e8-6e07d74476e5","Type":"ContainerDied","Data":"26a296f874f04ec3013d09210cb7a4b0d7a345059899395d1098b0988c01fd16"}
Sep 17 18:25:12.622544 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.622524    2562 scope.go:117] "RemoveContainer" containerID="f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23"
Sep 17 18:25:12.630257 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.630239    2562 scope.go:117] "RemoveContainer" containerID="94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d"
Sep 17 18:25:12.637016 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.636999    2562 scope.go:117] "RemoveContainer" containerID="ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29"
Sep 17 18:25:12.643415 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.643401    2562 scope.go:117] "RemoveContainer" containerID="f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0"
Sep 17 18:25:12.650948 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.650930    2562 scope.go:117] "RemoveContainer" containerID="f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23"
Sep 17 18:25:12.651234 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:12.651216    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23\": container with ID starting with f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23 not found: ID does not exist" containerID="f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23"
Sep 17 18:25:12.651307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.651269    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23"} err="failed to get container status \"f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23\": rpc error: code = NotFound desc = could not find container \"f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23\": container with ID starting with f4e5ccefc058c5e3676743fa217a7e5dd0fd99e7b9be5c3a28c7fa224399dd23 not found: ID does not exist"
Sep 17 18:25:12.651307 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.651294    2562 scope.go:117] "RemoveContainer" containerID="94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d"
Sep 17 18:25:12.651576 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:12.651559    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d\": container with ID starting with 94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d not found: ID does not exist" containerID="94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d"
Sep 17 18:25:12.651619 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.651582    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d"} err="failed to get container status \"94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d\": rpc error: code = NotFound desc = could not find container \"94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d\": container with ID starting with 94865dad9ef4e7fa2bd01c763b51762e3e838bace0c629e771eb4ee2d03b9f1d not found: ID does not exist"
Sep 17 18:25:12.651619 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.651598    2562 scope.go:117] "RemoveContainer" containerID="ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29"
Sep 17 18:25:12.651798 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:12.651779    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29\": container with ID starting with ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29 not found: ID does not exist" containerID="ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29"
Sep 17 18:25:12.651842 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.651803    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29"} err="failed to get container status \"ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29\": rpc error: code = NotFound desc = could not find container \"ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29\": container with ID starting with ca5dbe10d52502ddbca9c7a031b91e004615ed86d13375351ebee680582f1d29 not found: ID does not exist"
Sep 17 18:25:12.651842 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.651816    2562 scope.go:117] "RemoveContainer" containerID="f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0"
Sep 17 18:25:12.652088 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:12.652072    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0\": container with ID starting with f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0 not found: ID does not exist" containerID="f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0"
Sep 17 18:25:12.652129 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.652090    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0"} err="failed to get container status \"f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0\": rpc error: code = NotFound desc = could not find container \"f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0\": container with ID starting with f6f39c52fd454b37bfaa3ce2a24f6f6226ee3a475fc45c540f8e70c5d51d1aa0 not found: ID does not exist"
Sep 17 18:25:12.685385 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685360    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-cabundle-cert\") pod \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") "
Sep 17 18:25:12.685461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685413    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-rbac-proxy-sar-config\") pod \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") "
Sep 17 18:25:12.685461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685441    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wm9m\" (UniqueName: \"kubernetes.io/projected/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-api-access-4wm9m\") pod \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") "
Sep 17 18:25:12.685551 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685467    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kserve-provision-location\") pod \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") "
Sep 17 18:25:12.685589 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685550    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba3d394-bf0e-4258-84e8-6e07d74476e5-proxy-tls\") pod \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\" (UID: \"6ba3d394-bf0e-4258-84e8-6e07d74476e5\") "
Sep 17 18:25:12.685792 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685772    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ba3d394-bf0e-4258-84e8-6e07d74476e5" (UID: "6ba3d394-bf0e-4258-84e8-6e07d74476e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:12.685907 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685883    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ba3d394-bf0e-4258-84e8-6e07d74476e5" (UID: "6ba3d394-bf0e-4258-84e8-6e07d74476e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:12.685996 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.685943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ba3d394-bf0e-4258-84e8-6e07d74476e5" (UID: "6ba3d394-bf0e-4258-84e8-6e07d74476e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:12.687535 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.687510    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba3d394-bf0e-4258-84e8-6e07d74476e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ba3d394-bf0e-4258-84e8-6e07d74476e5" (UID: "6ba3d394-bf0e-4258-84e8-6e07d74476e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:12.687601 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.687536    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-api-access-4wm9m" (OuterVolumeSpecName: "kube-api-access-4wm9m") pod "6ba3d394-bf0e-4258-84e8-6e07d74476e5" (UID: "6ba3d394-bf0e-4258-84e8-6e07d74476e5"). InnerVolumeSpecName "kube-api-access-4wm9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:12.787308 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.787214    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ba3d394-bf0e-4258-84e8-6e07d74476e5-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:12.787308 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.787242    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:12.787308 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.787252    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:12.787308 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.787260    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wm9m\" (UniqueName: \"kubernetes.io/projected/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kube-api-access-4wm9m\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:12.787308 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.787269    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ba3d394-bf0e-4258-84e8-6e07d74476e5-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:12.946471 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.946442    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"]
Sep 17 18:25:12.950616 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:12.950591    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-545b85879c-8gtz7"]
Sep 17 18:25:13.276213 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.276188    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" path="/var/lib/kubelet/pods/6ba3d394-bf0e-4258-84e8-6e07d74476e5/volumes"
Sep 17 18:25:13.865351 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.865328    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:25:13.996880 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.996856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6465dc9-4305-4b94-b736-0310068182ba-kserve-provision-location\") pod \"f6465dc9-4305-4b94-b736-0310068182ba\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") "
Sep 17 18:25:13.997055 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.996941    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-cabundle-cert\") pod \"f6465dc9-4305-4b94-b736-0310068182ba\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") "
Sep 17 18:25:13.997055 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.996961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-kube-rbac-proxy-sar-config\") pod \"f6465dc9-4305-4b94-b736-0310068182ba\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") "
Sep 17 18:25:13.997055 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.996991    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls\") pod \"f6465dc9-4305-4b94-b736-0310068182ba\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") "
Sep 17 18:25:13.997055 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.997028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppl9h\" (UniqueName: \"kubernetes.io/projected/f6465dc9-4305-4b94-b736-0310068182ba-kube-api-access-ppl9h\") pod \"f6465dc9-4305-4b94-b736-0310068182ba\" (UID: \"f6465dc9-4305-4b94-b736-0310068182ba\") "
Sep 17 18:25:13.997286 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.997201    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6465dc9-4305-4b94-b736-0310068182ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6465dc9-4305-4b94-b736-0310068182ba" (UID: "f6465dc9-4305-4b94-b736-0310068182ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:13.997286 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.997278    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f6465dc9-4305-4b94-b736-0310068182ba" (UID: "f6465dc9-4305-4b94-b736-0310068182ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:13.997390 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.997335    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6465dc9-4305-4b94-b736-0310068182ba" (UID: "f6465dc9-4305-4b94-b736-0310068182ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:13.999077 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.999049    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f6465dc9-4305-4b94-b736-0310068182ba" (UID: "f6465dc9-4305-4b94-b736-0310068182ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:13.999208 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:13.999190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6465dc9-4305-4b94-b736-0310068182ba-kube-api-access-ppl9h" (OuterVolumeSpecName: "kube-api-access-ppl9h") pod "f6465dc9-4305-4b94-b736-0310068182ba" (UID: "f6465dc9-4305-4b94-b736-0310068182ba"). InnerVolumeSpecName "kube-api-access-ppl9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:14.097945 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.097894    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6465dc9-4305-4b94-b736-0310068182ba-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:14.097945 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.097941    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:14.097945 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.097952    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6465dc9-4305-4b94-b736-0310068182ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:14.098175 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.097960    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6465dc9-4305-4b94-b736-0310068182ba-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:14.098175 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.097970    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppl9h\" (UniqueName: \"kubernetes.io/projected/f6465dc9-4305-4b94-b736-0310068182ba-kube-api-access-ppl9h\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:14.631865 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.631838    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_531b7934-4fc4-4167-996d-2c20785450f3/storage-initializer/0.log"
Sep 17 18:25:14.632051 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.631883    2562 generic.go:356] "Generic (PLEG): container finished" podID="531b7934-4fc4-4167-996d-2c20785450f3" containerID="d7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430" exitCode=1
Sep 17 18:25:14.632051 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.631964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" event={"ID":"531b7934-4fc4-4167-996d-2c20785450f3","Type":"ContainerDied","Data":"d7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430"}
Sep 17 18:25:14.633978 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.633955    2562 generic.go:356] "Generic (PLEG): container finished" podID="f6465dc9-4305-4b94-b736-0310068182ba" containerID="10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4" exitCode=0
Sep 17 18:25:14.634092 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.634019    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"
Sep 17 18:25:14.634092 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.634041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerDied","Data":"10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4"}
Sep 17 18:25:14.634214 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.634102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c" event={"ID":"f6465dc9-4305-4b94-b736-0310068182ba","Type":"ContainerDied","Data":"48e73ad3ac7a7f31a79c1501330832b10e4ef9f14cf49888c03886b0e66853fc"}
Sep 17 18:25:14.634214 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.634142    2562 scope.go:117] "RemoveContainer" containerID="4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352"
Sep 17 18:25:14.644146 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.644129    2562 scope.go:117] "RemoveContainer" containerID="10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4"
Sep 17 18:25:14.658422 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.658404    2562 scope.go:117] "RemoveContainer" containerID="531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6"
Sep 17 18:25:14.665666 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.665642    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"]
Sep 17 18:25:14.669783 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.669762    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-7765d6c49d-drq4c"]
Sep 17 18:25:14.671545 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.671520    2562 scope.go:117] "RemoveContainer" containerID="4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352"
Sep 17 18:25:14.671962 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:14.671933    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352\": container with ID starting with 4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352 not found: ID does not exist" containerID="4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352"
Sep 17 18:25:14.672068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.671977    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352"} err="failed to get container status \"4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352\": rpc error: code = NotFound desc = could not find container \"4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352\": container with ID starting with 4ab3dd8f2ae6814b513cb9c9c9e7df77b0fc50768a19d9c1539e27867d5a6352 not found: ID does not exist"
Sep 17 18:25:14.672068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.672064    2562 scope.go:117] "RemoveContainer" containerID="10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4"
Sep 17 18:25:14.672411 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:14.672387    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4\": container with ID starting with 10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4 not found: ID does not exist" containerID="10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4"
Sep 17 18:25:14.672507 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.672417    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4"} err="failed to get container status \"10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4\": rpc error: code = NotFound desc = could not find container \"10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4\": container with ID starting with 10cf8a177f0801e61438eae8806f1472cae43cf1c4c0c0df7d732af8b7d723a4 not found: ID does not exist"
Sep 17 18:25:14.672507 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.672440    2562 scope.go:117] "RemoveContainer" containerID="531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6"
Sep 17 18:25:14.672754 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:14.672736    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6\": container with ID starting with 531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6 not found: ID does not exist" containerID="531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6"
Sep 17 18:25:14.672832 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:14.672758    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6"} err="failed to get container status \"531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6\": rpc error: code = NotFound desc = could not find container \"531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6\": container with ID starting with 531d5c1c97b9da1571f06ff5b52c0a2fa7b1f98b1bcdeafd5780adcea71df0f6 not found: ID does not exist"
Sep 17 18:25:15.276391 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:15.276360    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6465dc9-4305-4b94-b736-0310068182ba" path="/var/lib/kubelet/pods/f6465dc9-4305-4b94-b736-0310068182ba/volumes"
Sep 17 18:25:15.639829 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:15.639756    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_531b7934-4fc4-4167-996d-2c20785450f3/storage-initializer/0.log"
Sep 17 18:25:15.640007 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:15.639864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" event={"ID":"531b7934-4fc4-4167-996d-2c20785450f3","Type":"ContainerStarted","Data":"b00908ef605a9223e1233f0a465f1735fe3df89c4233557e4ee787a766567df5"}
Sep 17 18:25:18.652679 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:18.652651    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_531b7934-4fc4-4167-996d-2c20785450f3/storage-initializer/1.log"
Sep 17 18:25:18.653031 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:18.653023    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_531b7934-4fc4-4167-996d-2c20785450f3/storage-initializer/0.log"
Sep 17 18:25:18.653095 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:18.653053    2562 generic.go:356] "Generic (PLEG): container finished" podID="531b7934-4fc4-4167-996d-2c20785450f3" containerID="b00908ef605a9223e1233f0a465f1735fe3df89c4233557e4ee787a766567df5" exitCode=1
Sep 17 18:25:18.653095 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:18.653087    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" event={"ID":"531b7934-4fc4-4167-996d-2c20785450f3","Type":"ContainerDied","Data":"b00908ef605a9223e1233f0a465f1735fe3df89c4233557e4ee787a766567df5"}
Sep 17 18:25:18.653181 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:18.653142    2562 scope.go:117] "RemoveContainer" containerID="d7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430"
Sep 17 18:25:18.653466 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:18.653446    2562 scope.go:117] "RemoveContainer" containerID="d7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430"
Sep 17 18:25:18.663300 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:18.663275    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_kserve-ci-e2e-test(531b7934-4fc4-4167-996d-2c20785450f3)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" podUID="531b7934-4fc4-4167-996d-2c20785450f3"
Sep 17 18:25:18.663380 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:25:18.663340    2562 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_kserve-ci-e2e-test_531b7934-4fc4-4167-996d-2c20785450f3_0 in pod sandbox 0cbf0cd8f6d7ba3abe2eb6dab8d72c54233dbc838d6a5efb0ab37eec8504e65b from index: no such id: 'd7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430'" containerID="d7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430"
Sep 17 18:25:18.663427 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:18.663375    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_kserve-ci-e2e-test_531b7934-4fc4-4167-996d-2c20785450f3_0 in pod sandbox 0cbf0cd8f6d7ba3abe2eb6dab8d72c54233dbc838d6a5efb0ab37eec8504e65b from index: no such id: 'd7ef4e974aa1f433ed854bad2f0c428183ab97c88012fa85db17cb786fa7b430'"
Sep 17 18:25:19.658234 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:19.658205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_531b7934-4fc4-4167-996d-2c20785450f3/storage-initializer/1.log"
Sep 17 18:25:20.209957 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.209921    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"]
Sep 17 18:25:20.331291 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.331270    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_531b7934-4fc4-4167-996d-2c20785450f3/storage-initializer/1.log"
Sep 17 18:25:20.334555 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.334539    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:20.342361 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.342342    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls\") pod \"531b7934-4fc4-4167-996d-2c20785450f3\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") "
Sep 17 18:25:20.342487 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.342397    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/531b7934-4fc4-4167-996d-2c20785450f3-kserve-provision-location\") pod \"531b7934-4fc4-4167-996d-2c20785450f3\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") "
Sep 17 18:25:20.342487 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.342444    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/531b7934-4fc4-4167-996d-2c20785450f3-kube-rbac-proxy-sar-config\") pod \"531b7934-4fc4-4167-996d-2c20785450f3\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") "
Sep 17 18:25:20.342487 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.342474    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tll45\" (UniqueName: \"kubernetes.io/projected/531b7934-4fc4-4167-996d-2c20785450f3-kube-api-access-tll45\") pod \"531b7934-4fc4-4167-996d-2c20785450f3\" (UID: \"531b7934-4fc4-4167-996d-2c20785450f3\") "
Sep 17 18:25:20.342660 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.342612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531b7934-4fc4-4167-996d-2c20785450f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "531b7934-4fc4-4167-996d-2c20785450f3" (UID: "531b7934-4fc4-4167-996d-2c20785450f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:20.342779 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.342736    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531b7934-4fc4-4167-996d-2c20785450f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "531b7934-4fc4-4167-996d-2c20785450f3" (UID: "531b7934-4fc4-4167-996d-2c20785450f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:20.344364 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.344346    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531b7934-4fc4-4167-996d-2c20785450f3-kube-api-access-tll45" (OuterVolumeSpecName: "kube-api-access-tll45") pod "531b7934-4fc4-4167-996d-2c20785450f3" (UID: "531b7934-4fc4-4167-996d-2c20785450f3"). InnerVolumeSpecName "kube-api-access-tll45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:20.344513 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.344497    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "531b7934-4fc4-4167-996d-2c20785450f3" (UID: "531b7934-4fc4-4167-996d-2c20785450f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:20.443527 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.443502    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tll45\" (UniqueName: \"kubernetes.io/projected/531b7934-4fc4-4167-996d-2c20785450f3-kube-api-access-tll45\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.443527 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.443527    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/531b7934-4fc4-4167-996d-2c20785450f3-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.443718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.443541    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/531b7934-4fc4-4167-996d-2c20785450f3-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.443718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.443553    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/531b7934-4fc4-4167-996d-2c20785450f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:25:20.662850 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.662823    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg_531b7934-4fc4-4167-996d-2c20785450f3/storage-initializer/1.log"
Sep 17 18:25:20.663372 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.662881    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg" event={"ID":"531b7934-4fc4-4167-996d-2c20785450f3","Type":"ContainerDied","Data":"0cbf0cd8f6d7ba3abe2eb6dab8d72c54233dbc838d6a5efb0ab37eec8504e65b"}
Sep 17 18:25:20.663372 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.662935    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"
Sep 17 18:25:20.663372 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.662966    2562 scope.go:117] "RemoveContainer" containerID="b00908ef605a9223e1233f0a465f1735fe3df89c4233557e4ee787a766567df5"
Sep 17 18:25:20.700074 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.700046    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"]
Sep 17 18:25:20.706594 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:20.706569    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6fbcb7ffd-nmfhg"]
Sep 17 18:25:21.275756 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.275724    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="531b7934-4fc4-4167-996d-2c20785450f3" path="/var/lib/kubelet/pods/531b7934-4fc4-4167-996d-2c20785450f3/volumes"
Sep 17 18:25:21.372750 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.372709    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"]
Sep 17 18:25:21.373091 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373075    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="storage-initializer"
Sep 17 18:25:21.373142 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373095    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="storage-initializer"
Sep 17 18:25:21.373142 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373109    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531b7934-4fc4-4167-996d-2c20785450f3" containerName="storage-initializer"
Sep 17 18:25:21.373142 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373118    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="531b7934-4fc4-4167-996d-2c20785450f3" containerName="storage-initializer"
Sep 17 18:25:21.373142 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373131    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="531b7934-4fc4-4167-996d-2c20785450f3" containerName="storage-initializer"
Sep 17 18:25:21.373142 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373140    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="531b7934-4fc4-4167-996d-2c20785450f3" containerName="storage-initializer"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373153    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373162    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373173    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373181    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373196    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="storage-initializer"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373203    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="storage-initializer"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373221    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-agent"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373230    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-agent"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373240    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kube-rbac-proxy"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373248    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kube-rbac-proxy"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373258    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy"
Sep 17 18:25:21.373288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373266    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy"
Sep 17 18:25:21.373634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373333    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kube-rbac-proxy"
Sep 17 18:25:21.373634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373342    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-container"
Sep 17 18:25:21.373634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373350    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="531b7934-4fc4-4167-996d-2c20785450f3" containerName="storage-initializer"
Sep 17 18:25:21.373634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373358    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kserve-agent"
Sep 17 18:25:21.373634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373367    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba3d394-bf0e-4258-84e8-6e07d74476e5" containerName="kube-rbac-proxy"
Sep 17 18:25:21.373634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373374    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6465dc9-4305-4b94-b736-0310068182ba" containerName="kserve-container"
Sep 17 18:25:21.373634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.373499    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="531b7934-4fc4-4167-996d-2c20785450f3" containerName="storage-initializer"
Sep 17 18:25:21.378037 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.378015    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.381032 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.381013    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:25:21.381306 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.381291    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:25:21.381662 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.381645    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:25:21.381739 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.381661    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 18:25:21.381739 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.381651    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:25:21.382173 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.382156    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:25:21.382217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.382156    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:25:21.388773 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.388752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"]
Sep 17 18:25:21.450557 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.450528    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.450557 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.450558    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49755ab9-06c2-4050-a269-8067434ff4b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.450749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.450598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.450749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.450665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49755ab9-06c2-4050-a269-8067434ff4b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.450749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.450692    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbvcb\" (UniqueName: \"kubernetes.io/projected/49755ab9-06c2-4050-a269-8067434ff4b9-kube-api-access-qbvcb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.551961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.552010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49755ab9-06c2-4050-a269-8067434ff4b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552045 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.552031    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbvcb\" (UniqueName: \"kubernetes.io/projected/49755ab9-06c2-4050-a269-8067434ff4b9-kube-api-access-qbvcb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552295 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.552071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552295 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.552097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49755ab9-06c2-4050-a269-8067434ff4b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552511 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.552489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49755ab9-06c2-4050-a269-8067434ff4b9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552779 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.552757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.552816 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.552757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.559457 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.554952    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49755ab9-06c2-4050-a269-8067434ff4b9-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.563736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.563712    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbvcb\" (UniqueName: \"kubernetes.io/projected/49755ab9-06c2-4050-a269-8067434ff4b9-kube-api-access-qbvcb\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.688511 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.688487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:21.965845 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:21.965768    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"]
Sep 17 18:25:21.968723 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:25:21.968693    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49755ab9_06c2_4050_a269_8067434ff4b9.slice/crio-e921606a0d36bd101166a1f0542791ac3147ad74e19d75857167617efcbfd35e WatchSource:0}: Error finding container e921606a0d36bd101166a1f0542791ac3147ad74e19d75857167617efcbfd35e: Status 404 returned error can't find the container with id e921606a0d36bd101166a1f0542791ac3147ad74e19d75857167617efcbfd35e
Sep 17 18:25:22.672753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:22.672720    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerStarted","Data":"56b8bfe5301a9c28f7d43344d71383154a023e701016fe33f1c354907b50d88b"}
Sep 17 18:25:22.672949 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:22.672779    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerStarted","Data":"e921606a0d36bd101166a1f0542791ac3147ad74e19d75857167617efcbfd35e"}
Sep 17 18:25:23.677551 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:23.677523    2562 generic.go:356] "Generic (PLEG): container finished" podID="49755ab9-06c2-4050-a269-8067434ff4b9" containerID="56b8bfe5301a9c28f7d43344d71383154a023e701016fe33f1c354907b50d88b" exitCode=0
Sep 17 18:25:23.677943 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:23.677555    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerDied","Data":"56b8bfe5301a9c28f7d43344d71383154a023e701016fe33f1c354907b50d88b"}
Sep 17 18:25:24.683497 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:24.683465    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerStarted","Data":"400f3d944a2d12e80100a645b6e692a64c55824190d19aa49027f4d858bdaaa9"}
Sep 17 18:25:24.683856 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:24.683523    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerStarted","Data":"5424d33feec69d9fb92c16fd28b2fc86926afeab80784cb74dbf7f6cc2b8a362"}
Sep 17 18:25:24.683856 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:24.683613    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:24.703130 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:24.703087    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podStartSLOduration=2.6561041149999998 podStartE2EDuration="3.70307325s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.046969135s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:24.702397183 +0000 UTC m=+3015.889833032" watchObservedRunningTime="2026-09-17 18:25:24.70307325 +0000 UTC m=+3015.890509160"
Sep 17 18:25:25.686849 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:25.686819    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:25.687878 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:25.687855    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:25:26.689926 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:26.689878    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:25:31.694515 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:31.694488    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:25:31.694988 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:31.694938    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:25:41.695814 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:41.695778    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:25:51.696084 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:25:51.696017    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:26:01.695721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:01.695637    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:26:04.055930 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.055875    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"]
Sep 17 18:26:04.059447 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.059424    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.061844 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.061821    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 18:26:04.061987 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.061847    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:26:04.061987 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.061901    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:26:04.069745 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.069723    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"]
Sep 17 18:26:04.156577 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.156546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.156736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.156593    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.156736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.156611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ls2\" (UniqueName: \"kubernetes.io/projected/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-api-access-l8ls2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.156736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.156638    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.156736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.156672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258020 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.257987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258134 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.258026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ls2\" (UniqueName: \"kubernetes.io/projected/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-api-access-l8ls2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258134 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.258051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258134 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.258086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258265 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.258173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258308 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:26:04.258277    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 18:26:04.258401 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:26:04.258388    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls podName:acb3ae12-ffb6-4482-b4b1-1150bd6128b4 nodeName:}" failed. No retries permitted until 2026-09-17 18:26:04.758365695 +0000 UTC m=+3055.945801522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-mt92p" (UID: "acb3ae12-ffb6-4482-b4b1-1150bd6128b4") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Sep 17 18:26:04.258485 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.258470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258750 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.258733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.258821 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.258804    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.266822 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.266797    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ls2\" (UniqueName: \"kubernetes.io/projected/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-api-access-l8ls2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.763708 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.763675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.766153 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.766132    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-mt92p\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:04.969891 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:04.969858    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:05.241861 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:05.241836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"]
Sep 17 18:26:05.244351 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:26:05.244320    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb3ae12_ffb6_4482_b4b1_1150bd6128b4.slice/crio-fb6ebf051dc965f7b28d21fee6750a2fc6215f9cb3b2431d4354848d015a4122 WatchSource:0}: Error finding container fb6ebf051dc965f7b28d21fee6750a2fc6215f9cb3b2431d4354848d015a4122: Status 404 returned error can't find the container with id fb6ebf051dc965f7b28d21fee6750a2fc6215f9cb3b2431d4354848d015a4122
Sep 17 18:26:05.830328 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:05.830291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerStarted","Data":"1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089"}
Sep 17 18:26:05.830533 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:05.830367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerStarted","Data":"fb6ebf051dc965f7b28d21fee6750a2fc6215f9cb3b2431d4354848d015a4122"}
Sep 17 18:26:10.851112 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:10.851072    2562 generic.go:356] "Generic (PLEG): container finished" podID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerID="1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089" exitCode=0
Sep 17 18:26:10.851509 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:10.851130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerDied","Data":"1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089"}
Sep 17 18:26:10.852531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:10.852511    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:26:11.694953 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:11.694906    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:26:21.695683 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:21.695641    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:26:22.902305 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:22.902230    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerStarted","Data":"8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb"}
Sep 17 18:26:22.902305 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:22.902293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerStarted","Data":"bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811"}
Sep 17 18:26:22.902738 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:22.902547    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:22.902738 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:22.902687    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:22.903782 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:22.903758    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:26:22.923733 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:22.923695    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podStartSLOduration=2.5208302209999998 podStartE2EDuration="18.923682979s" totalImagesPullingTime="11.742271117s" totalInitContainerRuntime="4.660581641s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:22.921900466 +0000 UTC m=+3074.109336317" watchObservedRunningTime="2026-09-17 18:26:22.923682979 +0000 UTC m=+3074.111118827"
Sep 17 18:26:23.906294 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:23.906254    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:26:28.910347 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:28.910321    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:26:28.910793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:28.910768    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:26:31.695593 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:31.695555    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:26:38.911137 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:38.911099    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:26:41.696066 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:41.696037    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:26:48.910803 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:48.910764    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:26:51.373604 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:51.373567    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"]
Sep 17 18:26:51.374083 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:51.373931    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kube-rbac-proxy" containerID="cri-o://400f3d944a2d12e80100a645b6e692a64c55824190d19aa49027f4d858bdaaa9" gracePeriod=30
Sep 17 18:26:51.374083 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:51.373907    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" containerID="cri-o://5424d33feec69d9fb92c16fd28b2fc86926afeab80784cb74dbf7f6cc2b8a362" gracePeriod=30
Sep 17 18:26:51.691173 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:51.691085    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 17 18:26:51.695586 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:51.695556    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 17 18:26:52.007818 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:52.007784    2562 generic.go:356] "Generic (PLEG): container finished" podID="49755ab9-06c2-4050-a269-8067434ff4b9" containerID="400f3d944a2d12e80100a645b6e692a64c55824190d19aa49027f4d858bdaaa9" exitCode=2
Sep 17 18:26:52.008002 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:52.007827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerDied","Data":"400f3d944a2d12e80100a645b6e692a64c55824190d19aa49027f4d858bdaaa9"}
Sep 17 18:26:56.024887 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.024855    2562 generic.go:356] "Generic (PLEG): container finished" podID="49755ab9-06c2-4050-a269-8067434ff4b9" containerID="5424d33feec69d9fb92c16fd28b2fc86926afeab80784cb74dbf7f6cc2b8a362" exitCode=0
Sep 17 18:26:56.025301 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.024896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerDied","Data":"5424d33feec69d9fb92c16fd28b2fc86926afeab80784cb74dbf7f6cc2b8a362"}
Sep 17 18:26:56.379869 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.379840    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:26:56.456836 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.456799    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-kube-rbac-proxy-sar-config\") pod \"49755ab9-06c2-4050-a269-8067434ff4b9\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") "
Sep 17 18:26:56.456999 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.456846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49755ab9-06c2-4050-a269-8067434ff4b9-proxy-tls\") pod \"49755ab9-06c2-4050-a269-8067434ff4b9\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") "
Sep 17 18:26:56.456999 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.456887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49755ab9-06c2-4050-a269-8067434ff4b9-kserve-provision-location\") pod \"49755ab9-06c2-4050-a269-8067434ff4b9\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") "
Sep 17 18:26:56.456999 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.456952    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbvcb\" (UniqueName: \"kubernetes.io/projected/49755ab9-06c2-4050-a269-8067434ff4b9-kube-api-access-qbvcb\") pod \"49755ab9-06c2-4050-a269-8067434ff4b9\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") "
Sep 17 18:26:56.457131 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.457013    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-cabundle-cert\") pod \"49755ab9-06c2-4050-a269-8067434ff4b9\" (UID: \"49755ab9-06c2-4050-a269-8067434ff4b9\") "
Sep 17 18:26:56.457280 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.457239    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49755ab9-06c2-4050-a269-8067434ff4b9" (UID: "49755ab9-06c2-4050-a269-8067434ff4b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:56.457280 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.457267    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49755ab9-06c2-4050-a269-8067434ff4b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49755ab9-06c2-4050-a269-8067434ff4b9" (UID: "49755ab9-06c2-4050-a269-8067434ff4b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:56.457474 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.457453    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49755ab9-06c2-4050-a269-8067434ff4b9" (UID: "49755ab9-06c2-4050-a269-8067434ff4b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:56.458971 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.458946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49755ab9-06c2-4050-a269-8067434ff4b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49755ab9-06c2-4050-a269-8067434ff4b9" (UID: "49755ab9-06c2-4050-a269-8067434ff4b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:56.459112 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.459093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49755ab9-06c2-4050-a269-8067434ff4b9-kube-api-access-qbvcb" (OuterVolumeSpecName: "kube-api-access-qbvcb") pod "49755ab9-06c2-4050-a269-8067434ff4b9" (UID: "49755ab9-06c2-4050-a269-8067434ff4b9"). InnerVolumeSpecName "kube-api-access-qbvcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:56.557725 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.557665    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:26:56.557725 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.557685    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49755ab9-06c2-4050-a269-8067434ff4b9-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:26:56.557725 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.557695    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49755ab9-06c2-4050-a269-8067434ff4b9-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:26:56.557725 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.557704    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbvcb\" (UniqueName: \"kubernetes.io/projected/49755ab9-06c2-4050-a269-8067434ff4b9-kube-api-access-qbvcb\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:26:56.557725 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:56.557712    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49755ab9-06c2-4050-a269-8067434ff4b9-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:26:57.030802 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.030767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt" event={"ID":"49755ab9-06c2-4050-a269-8067434ff4b9","Type":"ContainerDied","Data":"e921606a0d36bd101166a1f0542791ac3147ad74e19d75857167617efcbfd35e"}
Sep 17 18:26:57.031228 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.030838    2562 scope.go:117] "RemoveContainer" containerID="400f3d944a2d12e80100a645b6e692a64c55824190d19aa49027f4d858bdaaa9"
Sep 17 18:26:57.031228 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.030778    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"
Sep 17 18:26:57.038835 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.038817    2562 scope.go:117] "RemoveContainer" containerID="5424d33feec69d9fb92c16fd28b2fc86926afeab80784cb74dbf7f6cc2b8a362"
Sep 17 18:26:57.046200 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.046179    2562 scope.go:117] "RemoveContainer" containerID="56b8bfe5301a9c28f7d43344d71383154a023e701016fe33f1c354907b50d88b"
Sep 17 18:26:57.052419 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.052391    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"]
Sep 17 18:26:57.055535 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.055514    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-74698c47d9-rncjt"]
Sep 17 18:26:57.276774 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:57.276745    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" path="/var/lib/kubelet/pods/49755ab9-06c2-4050-a269-8067434ff4b9/volumes"
Sep 17 18:26:58.910739 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:26:58.910704    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 18:27:03.667845 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.667805    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"]
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668103    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kube-rbac-proxy"
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668115    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kube-rbac-proxy"
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668132    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="storage-initializer"
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668139    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="storage-initializer"
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668149    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container"
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668156    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container"
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668199    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kserve-container"
Sep 17 18:27:03.668254 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.668211    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="49755ab9-06c2-4050-a269-8067434ff4b9" containerName="kube-rbac-proxy"
Sep 17 18:27:03.671575 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.671554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.674187 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.674169    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 18:27:03.674265 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.674173    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:27:03.683596 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.683574    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"]
Sep 17 18:27:03.710865 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.710826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.710865 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.710862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.711094 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.710881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.711094 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.710986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.711094 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.711016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxj7\" (UniqueName: \"kubernetes.io/projected/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-api-access-rdxj7\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.812195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.812158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.812195 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.812198    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxj7\" (UniqueName: \"kubernetes.io/projected/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-api-access-rdxj7\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.812442 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.812253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.812442 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.812272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.812442 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.812293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.812736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.812711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.813041 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.813021    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.813118 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.813042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.814567 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.814551    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.822156 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.822136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxj7\" (UniqueName: \"kubernetes.io/projected/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-api-access-rdxj7\") pod \"isvc-pmml-runtime-predictor-885fc8777-k49ph\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:03.981545 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:03.981471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:04.251534 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:04.251504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"]
Sep 17 18:27:04.253633 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:27:04.253601    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6e4cf9_4f89_45a2_a9ec_f82d31b2673b.slice/crio-168d080e72d1d668a2d9e50c1d1c500cfe33eb4df4680db60986472f6797ca0d WatchSource:0}: Error finding container 168d080e72d1d668a2d9e50c1d1c500cfe33eb4df4680db60986472f6797ca0d: Status 404 returned error can't find the container with id 168d080e72d1d668a2d9e50c1d1c500cfe33eb4df4680db60986472f6797ca0d
Sep 17 18:27:05.063968 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:05.063932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerStarted","Data":"e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc"}
Sep 17 18:27:05.064440 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:05.063988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerStarted","Data":"168d080e72d1d668a2d9e50c1d1c500cfe33eb4df4680db60986472f6797ca0d"}
Sep 17 18:27:08.911083 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:08.911046    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:27:09.079795 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:09.079764    2562 generic.go:356] "Generic (PLEG): container finished" podID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerID="e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc" exitCode=0
Sep 17 18:27:09.080013 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:09.079819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerDied","Data":"e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc"}
Sep 17 18:27:15.699783 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:15.699166    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"]
Sep 17 18:27:15.699783 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:15.699436    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container" containerID="cri-o://bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811" gracePeriod=30
Sep 17 18:27:15.699783 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:15.699597    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kube-rbac-proxy" containerID="cri-o://8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb" gracePeriod=30
Sep 17 18:27:17.117660 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:17.117623    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerStarted","Data":"02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd"}
Sep 17 18:27:17.118161 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:17.117691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerStarted","Data":"7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d"}
Sep 17 18:27:17.118161 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:17.117967    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:17.119788 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:17.119764    2562 generic.go:356] "Generic (PLEG): container finished" podID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerID="8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb" exitCode=2
Sep 17 18:27:17.119892 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:17.119794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerDied","Data":"8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb"}
Sep 17 18:27:17.137944 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:17.137884    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podStartSLOduration=2.9359077559999998 podStartE2EDuration="14.137870997s" totalImagesPullingTime="7.339458446s" totalInitContainerRuntime="3.862504795s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:17.137250371 +0000 UTC m=+3128.324686226" watchObservedRunningTime="2026-09-17 18:27:17.137870997 +0000 UTC m=+3128.325306846"
Sep 17 18:27:18.124115 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.124087    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:18.125501 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.125468    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:27:18.646051 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.646026    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:27:18.730028 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.729957    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls\") pod \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") "
Sep 17 18:27:18.730028 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.729997    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8ls2\" (UniqueName: \"kubernetes.io/projected/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-api-access-l8ls2\") pod \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") "
Sep 17 18:27:18.730028 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.730019    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-rbac-proxy-sar-config\") pod \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") "
Sep 17 18:27:18.730282 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.730039    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kserve-provision-location\") pod \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") "
Sep 17 18:27:18.730282 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.730063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-cabundle-cert\") pod \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\" (UID: \"acb3ae12-ffb6-4482-b4b1-1150bd6128b4\") "
Sep 17 18:27:18.730396 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.730372    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "acb3ae12-ffb6-4482-b4b1-1150bd6128b4" (UID: "acb3ae12-ffb6-4482-b4b1-1150bd6128b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:18.730396 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.730380    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "acb3ae12-ffb6-4482-b4b1-1150bd6128b4" (UID: "acb3ae12-ffb6-4482-b4b1-1150bd6128b4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:18.732126 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.732101    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "acb3ae12-ffb6-4482-b4b1-1150bd6128b4" (UID: "acb3ae12-ffb6-4482-b4b1-1150bd6128b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:18.732126 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.732108    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-api-access-l8ls2" (OuterVolumeSpecName: "kube-api-access-l8ls2") pod "acb3ae12-ffb6-4482-b4b1-1150bd6128b4" (UID: "acb3ae12-ffb6-4482-b4b1-1150bd6128b4"). InnerVolumeSpecName "kube-api-access-l8ls2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:18.741771 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.741748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acb3ae12-ffb6-4482-b4b1-1150bd6128b4" (UID: "acb3ae12-ffb6-4482-b4b1-1150bd6128b4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:18.830572 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.830544    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:27:18.830572 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.830568    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ls2\" (UniqueName: \"kubernetes.io/projected/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-api-access-l8ls2\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:27:18.830572 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.830578    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:27:18.830767 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.830586    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:27:18.830767 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:18.830596    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/acb3ae12-ffb6-4482-b4b1-1150bd6128b4-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:27:19.128957 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.128894    2562 generic.go:356] "Generic (PLEG): container finished" podID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerID="bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811" exitCode=0
Sep 17 18:27:19.129479 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.128970    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"
Sep 17 18:27:19.129479 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.128983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerDied","Data":"bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811"}
Sep 17 18:27:19.129479 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.129047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p" event={"ID":"acb3ae12-ffb6-4482-b4b1-1150bd6128b4","Type":"ContainerDied","Data":"fb6ebf051dc965f7b28d21fee6750a2fc6215f9cb3b2431d4354848d015a4122"}
Sep 17 18:27:19.129479 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.129075    2562 scope.go:117] "RemoveContainer" containerID="8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb"
Sep 17 18:27:19.129735 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.129710    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:27:19.138011 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.137996    2562 scope.go:117] "RemoveContainer" containerID="bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811"
Sep 17 18:27:19.145150 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.145135    2562 scope.go:117] "RemoveContainer" containerID="1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089"
Sep 17 18:27:19.151788 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.151774    2562 scope.go:117] "RemoveContainer" containerID="8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb"
Sep 17 18:27:19.152078 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:27:19.152061    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb\": container with ID starting with 8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb not found: ID does not exist" containerID="8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb"
Sep 17 18:27:19.152130 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.152085    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb"} err="failed to get container status \"8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb\": rpc error: code = NotFound desc = could not find container \"8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb\": container with ID starting with 8ada77dae484d605dd0260a4b39754b902f795590a6163a7d3c29b09bb77c2bb not found: ID does not exist"
Sep 17 18:27:19.152130 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.152099    2562 scope.go:117] "RemoveContainer" containerID="bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811"
Sep 17 18:27:19.152364 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:27:19.152344    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811\": container with ID starting with bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811 not found: ID does not exist" containerID="bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811"
Sep 17 18:27:19.152411 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.152370    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811"} err="failed to get container status \"bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811\": rpc error: code = NotFound desc = could not find container \"bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811\": container with ID starting with bf4ee014b10883a8f88fb07ef35739fe24c96a0bd9fda19d86e14b0c793c5811 not found: ID does not exist"
Sep 17 18:27:19.152411 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.152385    2562 scope.go:117] "RemoveContainer" containerID="1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089"
Sep 17 18:27:19.152635 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:27:19.152619    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089\": container with ID starting with 1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089 not found: ID does not exist" containerID="1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089"
Sep 17 18:27:19.152675 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.152639    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089"} err="failed to get container status \"1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089\": rpc error: code = NotFound desc = could not find container \"1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089\": container with ID starting with 1bd0c5da602099c1aa73c0361892270b41c867f9366faf1190ed936ba0af4089 not found: ID does not exist"
Sep 17 18:27:19.172753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.172719    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"]
Sep 17 18:27:19.185571 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.185550    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-mt92p"]
Sep 17 18:27:19.276566 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:19.276538    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" path="/var/lib/kubelet/pods/acb3ae12-ffb6-4482-b4b1-1150bd6128b4/volumes"
Sep 17 18:27:20.878938 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.878894    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"]
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879183    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container"
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879195    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container"
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879205    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kube-rbac-proxy"
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879211    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kube-rbac-proxy"
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879225    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="storage-initializer"
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879230    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="storage-initializer"
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879277    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kube-rbac-proxy"
Sep 17 18:27:20.879345 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.879284    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb3ae12-ffb6-4482-b4b1-1150bd6128b4" containerName="kserve-container"
Sep 17 18:27:20.883555 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.883540    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:20.885822 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.885795    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 18:27:20.885976 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.885859    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:27:20.893185 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.893165    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"]
Sep 17 18:27:20.944974 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.944940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:20.945139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.945028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:20.945139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.945064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:20.945139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.945114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e84b0f77-e655-4e55-8ebd-3a72009a0d47-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:20.945139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:20.945136    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlf2f\" (UniqueName: \"kubernetes.io/projected/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-api-access-tlf2f\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.046417 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.046378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.046607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.046433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.046607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.046457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.046607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.046496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e84b0f77-e655-4e55-8ebd-3a72009a0d47-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.046607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.046513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlf2f\" (UniqueName: \"kubernetes.io/projected/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-api-access-tlf2f\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.046847 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.046826    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.047303 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.047272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.047420 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.047271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.049156 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.049128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e84b0f77-e655-4e55-8ebd-3a72009a0d47-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.054954 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.054930    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlf2f\" (UniqueName: \"kubernetes.io/projected/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-api-access-tlf2f\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.194283 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.194202    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:21.468996 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:21.468969    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"]
Sep 17 18:27:21.471413 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:27:21.471384    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84b0f77_e655_4e55_8ebd_3a72009a0d47.slice/crio-1188a60f7aaddad6d6efaee3e7a7b5e709377651c4366f5144e5bbdf9c9cabe0 WatchSource:0}: Error finding container 1188a60f7aaddad6d6efaee3e7a7b5e709377651c4366f5144e5bbdf9c9cabe0: Status 404 returned error can't find the container with id 1188a60f7aaddad6d6efaee3e7a7b5e709377651c4366f5144e5bbdf9c9cabe0
Sep 17 18:27:22.142741 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:22.142701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerStarted","Data":"bd9ad17ec25f80fc187846d50d8ffa2d834ddf53d24a3ad1f1263ba8c7134459"}
Sep 17 18:27:22.143137 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:22.142780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerStarted","Data":"1188a60f7aaddad6d6efaee3e7a7b5e709377651c4366f5144e5bbdf9c9cabe0"}
Sep 17 18:27:24.134318 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:24.134284    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:27:24.134758 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:24.134732    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:27:27.161845 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:27.161813    2562 generic.go:356] "Generic (PLEG): container finished" podID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerID="bd9ad17ec25f80fc187846d50d8ffa2d834ddf53d24a3ad1f1263ba8c7134459" exitCode=0
Sep 17 18:27:27.162270 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:27.161858    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerDied","Data":"bd9ad17ec25f80fc187846d50d8ffa2d834ddf53d24a3ad1f1263ba8c7134459"}
Sep 17 18:27:28.168267 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:28.168233    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerStarted","Data":"d97f9c5e37b5777b5b0e89cf87305b8d4916dbde5bdd30dd0442a4dab003645d"}
Sep 17 18:27:28.168725 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:28.168293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerStarted","Data":"2127f13bf2429c3ce312bd900e67c87920fe70ea8327009adb5597e9b7c28e9d"}
Sep 17 18:27:28.168725 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:28.168619    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:28.168831 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:28.168733    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:28.169995 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:28.169970    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:27:28.187832 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:28.187786    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podStartSLOduration=2.786479338 podStartE2EDuration="8.1877726s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.401293262s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:27:28.186585406 +0000 UTC m=+3139.374021255" watchObservedRunningTime="2026-09-17 18:27:28.1877726 +0000 UTC m=+3139.375208448"
Sep 17 18:27:29.172108 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:29.172071    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:27:30.175533 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:30.175495    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:27:34.135412 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:34.135373    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:27:35.180978 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:35.180951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:27:35.181451 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:35.181424    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:27:44.134854 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:44.134815    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:27:45.181880 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:45.181837    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:27:54.135144 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:54.135098    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:27:55.181609 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:27:55.181578    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:28:04.134743 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:04.134701    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:28:05.181367 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:05.181330    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:28:14.135040 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:14.134998    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:28:15.182054 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:15.182028    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:28:22.578563 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:22.578528    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"]
Sep 17 18:28:22.578987 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:22.578759    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" containerID="cri-o://2127f13bf2429c3ce312bd900e67c87920fe70ea8327009adb5597e9b7c28e9d" gracePeriod=30
Sep 17 18:28:22.578987 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:22.578794    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kube-rbac-proxy" containerID="cri-o://d97f9c5e37b5777b5b0e89cf87305b8d4916dbde5bdd30dd0442a4dab003645d" gracePeriod=30
Sep 17 18:28:23.375793 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:23.375760    2562 generic.go:356] "Generic (PLEG): container finished" podID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerID="d97f9c5e37b5777b5b0e89cf87305b8d4916dbde5bdd30dd0442a4dab003645d" exitCode=2
Sep 17 18:28:23.376002 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:23.375836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerDied","Data":"d97f9c5e37b5777b5b0e89cf87305b8d4916dbde5bdd30dd0442a4dab003645d"}
Sep 17 18:28:24.134767 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:24.134728    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:28:25.176237 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.176198    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 17 18:28:25.182033 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.182007    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 18:28:25.384587 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.384554    2562 generic.go:356] "Generic (PLEG): container finished" podID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerID="2127f13bf2429c3ce312bd900e67c87920fe70ea8327009adb5597e9b7c28e9d" exitCode=0
Sep 17 18:28:25.384723 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.384606    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerDied","Data":"2127f13bf2429c3ce312bd900e67c87920fe70ea8327009adb5597e9b7c28e9d"}
Sep 17 18:28:25.416736 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.416714    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:28:25.524253 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.524223    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e84b0f77-e655-4e55-8ebd-3a72009a0d47-proxy-tls\") pod \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") "
Sep 17 18:28:25.524414 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.524296    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-cabundle-cert\") pod \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") "
Sep 17 18:28:25.524414 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.524354    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kserve-provision-location\") pod \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") "
Sep 17 18:28:25.524414 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.524379    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlf2f\" (UniqueName: \"kubernetes.io/projected/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-api-access-tlf2f\") pod \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") "
Sep 17 18:28:25.524588 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.524428    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-rbac-proxy-sar-config\") pod \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\" (UID: \"e84b0f77-e655-4e55-8ebd-3a72009a0d47\") "
Sep 17 18:28:25.524740 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.524711    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e84b0f77-e655-4e55-8ebd-3a72009a0d47" (UID: "e84b0f77-e655-4e55-8ebd-3a72009a0d47"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:25.524862 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.524749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e84b0f77-e655-4e55-8ebd-3a72009a0d47" (UID: "e84b0f77-e655-4e55-8ebd-3a72009a0d47"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:25.526272 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.526252    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84b0f77-e655-4e55-8ebd-3a72009a0d47-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e84b0f77-e655-4e55-8ebd-3a72009a0d47" (UID: "e84b0f77-e655-4e55-8ebd-3a72009a0d47"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:25.526379 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.526358    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-api-access-tlf2f" (OuterVolumeSpecName: "kube-api-access-tlf2f") pod "e84b0f77-e655-4e55-8ebd-3a72009a0d47" (UID: "e84b0f77-e655-4e55-8ebd-3a72009a0d47"). InnerVolumeSpecName "kube-api-access-tlf2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:25.534391 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.534364    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e84b0f77-e655-4e55-8ebd-3a72009a0d47" (UID: "e84b0f77-e655-4e55-8ebd-3a72009a0d47"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:25.624894 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.624864    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:25.624894 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.624891    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:25.625068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.624906    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlf2f\" (UniqueName: \"kubernetes.io/projected/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-api-access-tlf2f\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:25.625068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.624942    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e84b0f77-e655-4e55-8ebd-3a72009a0d47-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:25.625068 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:25.624956    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e84b0f77-e655-4e55-8ebd-3a72009a0d47-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:26.389791 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:26.389766    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"
Sep 17 18:28:26.390278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:26.389755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k" event={"ID":"e84b0f77-e655-4e55-8ebd-3a72009a0d47","Type":"ContainerDied","Data":"1188a60f7aaddad6d6efaee3e7a7b5e709377651c4366f5144e5bbdf9c9cabe0"}
Sep 17 18:28:26.390278 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:26.389929    2562 scope.go:117] "RemoveContainer" containerID="d97f9c5e37b5777b5b0e89cf87305b8d4916dbde5bdd30dd0442a4dab003645d"
Sep 17 18:28:26.398235 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:26.398217    2562 scope.go:117] "RemoveContainer" containerID="2127f13bf2429c3ce312bd900e67c87920fe70ea8327009adb5597e9b7c28e9d"
Sep 17 18:28:26.405429 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:26.405409    2562 scope.go:117] "RemoveContainer" containerID="bd9ad17ec25f80fc187846d50d8ffa2d834ddf53d24a3ad1f1263ba8c7134459"
Sep 17 18:28:26.411998 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:26.411974    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"]
Sep 17 18:28:26.414596 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:26.414575    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-d7b2k"]
Sep 17 18:28:27.276354 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.276317    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" path="/var/lib/kubelet/pods/e84b0f77-e655-4e55-8ebd-3a72009a0d47/volumes"
Sep 17 18:28:27.855879 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.855844    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"]
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856115    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="storage-initializer"
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856127    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="storage-initializer"
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856137    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container"
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856143    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container"
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856152    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kube-rbac-proxy"
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856158    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kube-rbac-proxy"
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856201    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kube-rbac-proxy"
Sep 17 18:28:27.856258 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.856210    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e84b0f77-e655-4e55-8ebd-3a72009a0d47" containerName="kserve-container"
Sep 17 18:28:27.860648 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.860628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:27.863683 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.863664    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 18:28:27.863999 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.863984    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 18:28:27.898549 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.898518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"]
Sep 17 18:28:27.940192 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.940160    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11c4b90-647e-439f-a516-06810755fc93-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:27.940371 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.940197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:27.940371 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.940222    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b11c4b90-647e-439f-a516-06810755fc93-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:27.940371 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.940266    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:27.940371 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:27.940301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54ds\" (UniqueName: \"kubernetes.io/projected/b11c4b90-647e-439f-a516-06810755fc93-kube-api-access-z54ds\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.040957 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.040902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.040957 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.040960    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54ds\" (UniqueName: \"kubernetes.io/projected/b11c4b90-647e-439f-a516-06810755fc93-kube-api-access-z54ds\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.041217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.041018    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11c4b90-647e-439f-a516-06810755fc93-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.041217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.041034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.041217 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.041055    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b11c4b90-647e-439f-a516-06810755fc93-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.041460 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.041434    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b11c4b90-647e-439f-a516-06810755fc93-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.041619 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.041598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.041704 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.041683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.043447 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.043415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11c4b90-647e-439f-a516-06810755fc93-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.050634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.050605    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54ds\" (UniqueName: \"kubernetes.io/projected/b11c4b90-647e-439f-a516-06810755fc93-kube-api-access-z54ds\") pod \"isvc-pmml-predictor-8674795f59-cm79d\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.171201 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.171105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:28.444216 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:28.444127    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"]
Sep 17 18:28:28.448970 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:28:28.448568    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11c4b90_647e_439f_a516_06810755fc93.slice/crio-17b0b64e977db951bc5950cd204884d93716f9e93faf8668855a5ee00a996889 WatchSource:0}: Error finding container 17b0b64e977db951bc5950cd204884d93716f9e93faf8668855a5ee00a996889: Status 404 returned error can't find the container with id 17b0b64e977db951bc5950cd204884d93716f9e93faf8668855a5ee00a996889
Sep 17 18:28:29.402035 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:29.402001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerStarted","Data":"798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67"}
Sep 17 18:28:29.402499 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:29.402071    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerStarted","Data":"17b0b64e977db951bc5950cd204884d93716f9e93faf8668855a5ee00a996889"}
Sep 17 18:28:30.272328 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:30.272286    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:28:32.413382 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:32.413301    2562 generic.go:356] "Generic (PLEG): container finished" podID="b11c4b90-647e-439f-a516-06810755fc93" containerID="798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67" exitCode=0
Sep 17 18:28:32.413382 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:32.413363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerDied","Data":"798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67"}
Sep 17 18:28:33.418784 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:33.418753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerStarted","Data":"85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df"}
Sep 17 18:28:33.419214 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:33.418811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerStarted","Data":"2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074"}
Sep 17 18:28:33.419214 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:33.419009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:33.439823 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:33.439780    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podStartSLOduration=2.804204148 podStartE2EDuration="6.439766883s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.635562735s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:33.438303849 +0000 UTC m=+3204.625739698" watchObservedRunningTime="2026-09-17 18:28:33.439766883 +0000 UTC m=+3204.627202787"
Sep 17 18:28:34.422566 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:34.422535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:34.423849 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:34.423821    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:28:35.426215 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:35.426177    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:28:40.272745 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:40.272703    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 17 18:28:40.431123 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:40.431090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:28:40.431651 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:40.431625    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:28:50.272886 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:50.272852    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:28:50.431714 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:50.431677    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:28:55.255261 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:55.255226    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"]
Sep 17 18:28:55.255718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:55.255549    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container" containerID="cri-o://7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d" gracePeriod=30
Sep 17 18:28:55.255718 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:55.255555    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kube-rbac-proxy" containerID="cri-o://02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd" gracePeriod=30
Sep 17 18:28:55.500636 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:55.500608    2562 generic.go:356] "Generic (PLEG): container finished" podID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerID="02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd" exitCode=2
Sep 17 18:28:55.500788 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:55.500686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerDied","Data":"02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd"}
Sep 17 18:28:59.129580 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.129494    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 17 18:28:59.285269 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.285244    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:28:59.371131 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371096    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdxj7\" (UniqueName: \"kubernetes.io/projected/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-api-access-rdxj7\") pod \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") "
Sep 17 18:28:59.371275 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371173    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-cabundle-cert\") pod \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") "
Sep 17 18:28:59.371275 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-rbac-proxy-sar-config\") pod \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") "
Sep 17 18:28:59.371275 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-proxy-tls\") pod \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") "
Sep 17 18:28:59.371275 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kserve-provision-location\") pod \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\" (UID: \"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b\") "
Sep 17 18:28:59.371637 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" (UID: "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:59.371637 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371620    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" (UID: "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:59.371826 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.371628    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" (UID: "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:59.373331 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.373304    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-api-access-rdxj7" (OuterVolumeSpecName: "kube-api-access-rdxj7") pod "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" (UID: "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b"). InnerVolumeSpecName "kube-api-access-rdxj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:59.373331 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.373315    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" (UID: "da6e4cf9-4f89-45a2-a9ec-f82d31b2673b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:59.472495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.472432    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:59.472495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.472456    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdxj7\" (UniqueName: \"kubernetes.io/projected/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-api-access-rdxj7\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:59.472495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.472466    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:59.472495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.472476    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:59.472495 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.472485    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:28:59.521393 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.521354    2562 generic.go:356] "Generic (PLEG): container finished" podID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerID="7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d" exitCode=0
Sep 17 18:28:59.521524 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.521489    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerDied","Data":"7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d"}
Sep 17 18:28:59.521524 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.521514    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"
Sep 17 18:28:59.521640 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.521612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph" event={"ID":"da6e4cf9-4f89-45a2-a9ec-f82d31b2673b","Type":"ContainerDied","Data":"168d080e72d1d668a2d9e50c1d1c500cfe33eb4df4680db60986472f6797ca0d"}
Sep 17 18:28:59.521692 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.521643    2562 scope.go:117] "RemoveContainer" containerID="02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd"
Sep 17 18:28:59.530608 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.530587    2562 scope.go:117] "RemoveContainer" containerID="7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d"
Sep 17 18:28:59.537607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.537589    2562 scope.go:117] "RemoveContainer" containerID="e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc"
Sep 17 18:28:59.543310 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.543289    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"]
Sep 17 18:28:59.544486 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.544471    2562 scope.go:117] "RemoveContainer" containerID="02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd"
Sep 17 18:28:59.544745 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:28:59.544729    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd\": container with ID starting with 02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd not found: ID does not exist" containerID="02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd"
Sep 17 18:28:59.544796 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.544752    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd"} err="failed to get container status \"02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd\": rpc error: code = NotFound desc = could not find container \"02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd\": container with ID starting with 02eb46dc1715e4cc459099eaa15286f055233132857fc4b71d52fdfca19e64cd not found: ID does not exist"
Sep 17 18:28:59.544796 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.544770    2562 scope.go:117] "RemoveContainer" containerID="7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d"
Sep 17 18:28:59.545030 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:28:59.545010    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d\": container with ID starting with 7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d not found: ID does not exist" containerID="7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d"
Sep 17 18:28:59.545080 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.545037    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d"} err="failed to get container status \"7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d\": rpc error: code = NotFound desc = could not find container \"7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d\": container with ID starting with 7d8c1ca8a029aa7d4f2e1e322901a767197d526b5dfeff07a82b86737915709d not found: ID does not exist"
Sep 17 18:28:59.545080 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.545052    2562 scope.go:117] "RemoveContainer" containerID="e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc"
Sep 17 18:28:59.545297 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:28:59.545280    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc\": container with ID starting with e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc not found: ID does not exist" containerID="e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc"
Sep 17 18:28:59.545339 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.545301    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc"} err="failed to get container status \"e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc\": rpc error: code = NotFound desc = could not find container \"e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc\": container with ID starting with e437868e993b26abbc6480e8e72fed20c8b326141e50348206c6cc8eec6590dc not found: ID does not exist"
Sep 17 18:28:59.550265 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:28:59.550243    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-k49ph"]
Sep 17 18:29:00.432603 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.432568    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:29:00.449666 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.449634    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"]
Sep 17 18:29:00.449952 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.449939    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kube-rbac-proxy"
Sep 17 18:29:00.450008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.449955    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kube-rbac-proxy"
Sep 17 18:29:00.450008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.449969    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container"
Sep 17 18:29:00.450008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.449974    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container"
Sep 17 18:29:00.450008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.449982    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="storage-initializer"
Sep 17 18:29:00.450008 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.449988    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="storage-initializer"
Sep 17 18:29:00.450170 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.450044    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kserve-container"
Sep 17 18:29:00.450170 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.450054    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" containerName="kube-rbac-proxy"
Sep 17 18:29:00.454208 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.454192    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.456314 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.456289    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 18:29:00.456425 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.456377    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:29:00.462833 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.462813    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"]
Sep 17 18:29:00.580754 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.580716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.580936 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.580791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.580936 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.580834    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzxb5\" (UniqueName: \"kubernetes.io/projected/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-api-access-vzxb5\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.580936 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.580885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.580936 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.580908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2fd2e56-6877-4c31-a70a-62bd1c037429-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.682282 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.682247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.682282 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.682281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzxb5\" (UniqueName: \"kubernetes.io/projected/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-api-access-vzxb5\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.682496 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.682303    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.682496 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.682321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2fd2e56-6877-4c31-a70a-62bd1c037429-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.682496 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.682359    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.682496 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:29:00.682411    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 18:29:00.682496 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:29:00.682485    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls podName:f2fd2e56-6877-4c31-a70a-62bd1c037429 nodeName:}" failed. No retries permitted until 2026-09-17 18:29:01.182465248 +0000 UTC m=+3232.369901091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" (UID: "f2fd2e56-6877-4c31-a70a-62bd1c037429") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 18:29:00.682827 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.682767    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2fd2e56-6877-4c31-a70a-62bd1c037429-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.682972 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.682954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.683015 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.683003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:00.691035 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:00.691006    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzxb5\" (UniqueName: \"kubernetes.io/projected/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-api-access-vzxb5\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:01.187237 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:01.187201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:01.189583 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:01.189562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:01.276763 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:01.276736    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6e4cf9-4f89-45a2-a9ec-f82d31b2673b" path="/var/lib/kubelet/pods/da6e4cf9-4f89-45a2-a9ec-f82d31b2673b/volumes"
Sep 17 18:29:01.364665 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:01.364633    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:01.638050 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:01.638024    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"]
Sep 17 18:29:02.534293 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:02.534260    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerStarted","Data":"7e613bf38eb6b3d56ff7bfd0d8a2ecc522adeaca27eac36e65de77d6a1543315"}
Sep 17 18:29:02.534460 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:02.534317    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerStarted","Data":"98d5c442f394b24bce95f0e56f63f4db6ee71e176c542007a324cd20e9d0e037"}
Sep 17 18:29:05.546986 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:05.546951    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerID="7e613bf38eb6b3d56ff7bfd0d8a2ecc522adeaca27eac36e65de77d6a1543315" exitCode=0
Sep 17 18:29:05.547391 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:05.546991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerDied","Data":"7e613bf38eb6b3d56ff7bfd0d8a2ecc522adeaca27eac36e65de77d6a1543315"}
Sep 17 18:29:06.554072 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:06.554033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerStarted","Data":"51adfd7a895532337eb988f3572b564715aca04fd5833629f05878d01450ee9d"}
Sep 17 18:29:06.554549 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:06.554113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerStarted","Data":"07dfd33a2ef4e90e58461b298c3eda2ec981c4884d4f328469a532722cd207ba"}
Sep 17 18:29:06.554549 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:06.554418    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:06.554549 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:06.554486    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:06.555946 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:06.555897    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:29:06.575357 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:06.575312    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podStartSLOduration=2.845560085 podStartE2EDuration="6.575298831s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.729738746s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:06.573540375 +0000 UTC m=+3237.760976223" watchObservedRunningTime="2026-09-17 18:29:06.575298831 +0000 UTC m=+3237.762734680"
Sep 17 18:29:07.558232 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:07.558189    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:29:10.431770 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:10.431730    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:29:12.562513 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:12.562485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:29:12.562974 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:12.562950    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:29:20.432018 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:20.431979    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:29:22.563607 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:22.563563    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:29:30.431727 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:30.431691    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:29:32.563709 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:32.563671    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:29:40.432128 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:40.432082    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:29:42.563057 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:42.563016    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:29:49.273116 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:49.273070    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 18:29:52.563402 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:52.563366    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:29:59.276475 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:29:59.276446    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:30:02.563006 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:02.562969    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:30:09.577850 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:09.577811    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"]
Sep 17 18:30:09.578262 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:09.578070    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container" containerID="cri-o://2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074" gracePeriod=30
Sep 17 18:30:09.578262 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:09.578150    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kube-rbac-proxy" containerID="cri-o://85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df" gracePeriod=30
Sep 17 18:30:09.786161 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:09.786127    2562 generic.go:356] "Generic (PLEG): container finished" podID="b11c4b90-647e-439f-a516-06810755fc93" containerID="85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df" exitCode=2
Sep 17 18:30:09.786320 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:09.786186    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerDied","Data":"85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df"}
Sep 17 18:30:10.426615 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:10.426572    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 17 18:30:12.563500 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:12.563461    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:30:13.613999 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.613970    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:30:13.684309 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684270    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b11c4b90-647e-439f-a516-06810755fc93-kserve-provision-location\") pod \"b11c4b90-647e-439f-a516-06810755fc93\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") "
Sep 17 18:30:13.684473 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684325    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11c4b90-647e-439f-a516-06810755fc93-proxy-tls\") pod \"b11c4b90-647e-439f-a516-06810755fc93\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") "
Sep 17 18:30:13.684473 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684353    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-cabundle-cert\") pod \"b11c4b90-647e-439f-a516-06810755fc93\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") "
Sep 17 18:30:13.684473 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z54ds\" (UniqueName: \"kubernetes.io/projected/b11c4b90-647e-439f-a516-06810755fc93-kube-api-access-z54ds\") pod \"b11c4b90-647e-439f-a516-06810755fc93\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") "
Sep 17 18:30:13.684473 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684393    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-kube-rbac-proxy-sar-config\") pod \"b11c4b90-647e-439f-a516-06810755fc93\" (UID: \"b11c4b90-647e-439f-a516-06810755fc93\") "
Sep 17 18:30:13.684711 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684531    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11c4b90-647e-439f-a516-06810755fc93-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b11c4b90-647e-439f-a516-06810755fc93" (UID: "b11c4b90-647e-439f-a516-06810755fc93"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:13.684779 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684747    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b11c4b90-647e-439f-a516-06810755fc93" (UID: "b11c4b90-647e-439f-a516-06810755fc93"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:13.684824 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.684795    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b11c4b90-647e-439f-a516-06810755fc93" (UID: "b11c4b90-647e-439f-a516-06810755fc93"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:13.686511 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.686482    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11c4b90-647e-439f-a516-06810755fc93-kube-api-access-z54ds" (OuterVolumeSpecName: "kube-api-access-z54ds") pod "b11c4b90-647e-439f-a516-06810755fc93" (UID: "b11c4b90-647e-439f-a516-06810755fc93"). InnerVolumeSpecName "kube-api-access-z54ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:13.686613 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.686560    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11c4b90-647e-439f-a516-06810755fc93-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b11c4b90-647e-439f-a516-06810755fc93" (UID: "b11c4b90-647e-439f-a516-06810755fc93"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:13.785757 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.785734    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:13.785757 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.785756    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54ds\" (UniqueName: \"kubernetes.io/projected/b11c4b90-647e-439f-a516-06810755fc93-kube-api-access-z54ds\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:13.785906 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.785766    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b11c4b90-647e-439f-a516-06810755fc93-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:13.785906 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.785778    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b11c4b90-647e-439f-a516-06810755fc93-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:13.785906 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.785788    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b11c4b90-647e-439f-a516-06810755fc93-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:13.802487 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.802460    2562 generic.go:356] "Generic (PLEG): container finished" podID="b11c4b90-647e-439f-a516-06810755fc93" containerID="2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074" exitCode=0
Sep 17 18:30:13.802588 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.802514    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"
Sep 17 18:30:13.802588 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.802539    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerDied","Data":"2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074"}
Sep 17 18:30:13.802663 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.802595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d" event={"ID":"b11c4b90-647e-439f-a516-06810755fc93","Type":"ContainerDied","Data":"17b0b64e977db951bc5950cd204884d93716f9e93faf8668855a5ee00a996889"}
Sep 17 18:30:13.802663 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.802621    2562 scope.go:117] "RemoveContainer" containerID="85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df"
Sep 17 18:30:13.810639 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.810624    2562 scope.go:117] "RemoveContainer" containerID="2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074"
Sep 17 18:30:13.819732 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.819715    2562 scope.go:117] "RemoveContainer" containerID="798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67"
Sep 17 18:30:13.823535 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.823514    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"]
Sep 17 18:30:13.826757 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.826741    2562 scope.go:117] "RemoveContainer" containerID="85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df"
Sep 17 18:30:13.827119 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:30:13.827100    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df\": container with ID starting with 85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df not found: ID does not exist" containerID="85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df"
Sep 17 18:30:13.827182 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.827126    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df"} err="failed to get container status \"85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df\": rpc error: code = NotFound desc = could not find container \"85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df\": container with ID starting with 85a0573c97f924d7a2dba4ce814608f4e8b6e0d00aed82d8eb7544af619ec3df not found: ID does not exist"
Sep 17 18:30:13.827182 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.827142    2562 scope.go:117] "RemoveContainer" containerID="2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074"
Sep 17 18:30:13.827569 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:30:13.827523    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074\": container with ID starting with 2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074 not found: ID does not exist" containerID="2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074"
Sep 17 18:30:13.827569 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.827555    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074"} err="failed to get container status \"2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074\": rpc error: code = NotFound desc = could not find container \"2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074\": container with ID starting with 2086f1873b5b15b34ca42d5f32737dd75df7cb2d0a898f19e32c76bdead2e074 not found: ID does not exist"
Sep 17 18:30:13.827721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.827576    2562 scope.go:117] "RemoveContainer" containerID="798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67"
Sep 17 18:30:13.827949 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:30:13.827875    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67\": container with ID starting with 798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67 not found: ID does not exist" containerID="798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67"
Sep 17 18:30:13.828418 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.827901    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67"} err="failed to get container status \"798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67\": rpc error: code = NotFound desc = could not find container \"798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67\": container with ID starting with 798680ce6c8bf02f7d8e4030ee48428956b13dbca7c69de0bd624b88a72b2e67 not found: ID does not exist"
Sep 17 18:30:13.829546 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:13.829528    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-cm79d"]
Sep 17 18:30:15.275634 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:15.275589    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11c4b90-647e-439f-a516-06810755fc93" path="/var/lib/kubelet/pods/b11c4b90-647e-439f-a516-06810755fc93/volumes"
Sep 17 18:30:20.271758 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:20.271715    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:30:30.272502 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:30.272416    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 17 18:30:40.272515 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:40.272476    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:30:52.095302 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:52.095271    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"]
Sep 17 18:30:52.095697 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:52.095507    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container" containerID="cri-o://07dfd33a2ef4e90e58461b298c3eda2ec981c4884d4f328469a532722cd207ba" gracePeriod=30
Sep 17 18:30:52.095697 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:52.095531    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kube-rbac-proxy" containerID="cri-o://51adfd7a895532337eb988f3572b564715aca04fd5833629f05878d01450ee9d" gracePeriod=30
Sep 17 18:30:52.559401 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:52.559356    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 17 18:30:52.944377 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:52.942353    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerID="51adfd7a895532337eb988f3572b564715aca04fd5833629f05878d01450ee9d" exitCode=2
Sep 17 18:30:52.944377 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:52.942438    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerDied","Data":"51adfd7a895532337eb988f3572b564715aca04fd5833629f05878d01450ee9d"}
Sep 17 18:30:55.955292 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:55.955260    2562 generic.go:356] "Generic (PLEG): container finished" podID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerID="07dfd33a2ef4e90e58461b298c3eda2ec981c4884d4f328469a532722cd207ba" exitCode=0
Sep 17 18:30:55.955647 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:55.955334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerDied","Data":"07dfd33a2ef4e90e58461b298c3eda2ec981c4884d4f328469a532722cd207ba"}
Sep 17 18:30:56.130627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.130605    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:30:56.187721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.187690    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-rbac-proxy-sar-config\") pod \"f2fd2e56-6877-4c31-a70a-62bd1c037429\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") "
Sep 17 18:30:56.187721 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.187725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls\") pod \"f2fd2e56-6877-4c31-a70a-62bd1c037429\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") "
Sep 17 18:30:56.187893 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.187758    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzxb5\" (UniqueName: \"kubernetes.io/projected/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-api-access-vzxb5\") pod \"f2fd2e56-6877-4c31-a70a-62bd1c037429\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") "
Sep 17 18:30:56.187969 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.187887    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2fd2e56-6877-4c31-a70a-62bd1c037429-kserve-provision-location\") pod \"f2fd2e56-6877-4c31-a70a-62bd1c037429\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") "
Sep 17 18:30:56.188030 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.187967    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-cabundle-cert\") pod \"f2fd2e56-6877-4c31-a70a-62bd1c037429\" (UID: \"f2fd2e56-6877-4c31-a70a-62bd1c037429\") "
Sep 17 18:30:56.188170 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.188149    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2fd2e56-6877-4c31-a70a-62bd1c037429" (UID: "f2fd2e56-6877-4c31-a70a-62bd1c037429"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:56.188288 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.188258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fd2e56-6877-4c31-a70a-62bd1c037429-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2fd2e56-6877-4c31-a70a-62bd1c037429" (UID: "f2fd2e56-6877-4c31-a70a-62bd1c037429"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:56.188397 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.188323    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2fd2e56-6877-4c31-a70a-62bd1c037429" (UID: "f2fd2e56-6877-4c31-a70a-62bd1c037429"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:56.189844 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.189824    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2fd2e56-6877-4c31-a70a-62bd1c037429" (UID: "f2fd2e56-6877-4c31-a70a-62bd1c037429"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:56.189950 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.189863    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-api-access-vzxb5" (OuterVolumeSpecName: "kube-api-access-vzxb5") pod "f2fd2e56-6877-4c31-a70a-62bd1c037429" (UID: "f2fd2e56-6877-4c31-a70a-62bd1c037429"). InnerVolumeSpecName "kube-api-access-vzxb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:56.288627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.288561    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:56.288627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.288582    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2fd2e56-6877-4c31-a70a-62bd1c037429-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:56.288627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.288593    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzxb5\" (UniqueName: \"kubernetes.io/projected/f2fd2e56-6877-4c31-a70a-62bd1c037429-kube-api-access-vzxb5\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:56.288627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.288601    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2fd2e56-6877-4c31-a70a-62bd1c037429-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:56.288627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.288609    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2fd2e56-6877-4c31-a70a-62bd1c037429-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:30:56.960589 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.960552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt" event={"ID":"f2fd2e56-6877-4c31-a70a-62bd1c037429","Type":"ContainerDied","Data":"98d5c442f394b24bce95f0e56f63f4db6ee71e176c542007a324cd20e9d0e037"}
Sep 17 18:30:56.960996 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.960605    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"
Sep 17 18:30:56.960996 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.960633    2562 scope.go:117] "RemoveContainer" containerID="51adfd7a895532337eb988f3572b564715aca04fd5833629f05878d01450ee9d"
Sep 17 18:30:56.970161 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.970143    2562 scope.go:117] "RemoveContainer" containerID="07dfd33a2ef4e90e58461b298c3eda2ec981c4884d4f328469a532722cd207ba"
Sep 17 18:30:56.977076 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.977059    2562 scope.go:117] "RemoveContainer" containerID="7e613bf38eb6b3d56ff7bfd0d8a2ecc522adeaca27eac36e65de77d6a1543315"
Sep 17 18:30:56.982609 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.982590    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"]
Sep 17 18:30:56.986519 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:56.986499    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-4q4kt"]
Sep 17 18:30:57.275844 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.275817    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" path="/var/lib/kubelet/pods/f2fd2e56-6877-4c31-a70a-62bd1c037429/volumes"
Sep 17 18:30:57.290673 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290645    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"]
Sep 17 18:30:57.290891 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290877    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="storage-initializer"
Sep 17 18:30:57.290960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290893    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="storage-initializer"
Sep 17 18:30:57.290960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290907    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="storage-initializer"
Sep 17 18:30:57.290960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290928    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="storage-initializer"
Sep 17 18:30:57.290960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290938    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kube-rbac-proxy"
Sep 17 18:30:57.290960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290947    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kube-rbac-proxy"
Sep 17 18:30:57.290960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290959    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kube-rbac-proxy"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290969    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kube-rbac-proxy"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290979    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290985    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290993    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.290998    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.291048    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kserve-container"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.291056    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kserve-container"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.291062    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2fd2e56-6877-4c31-a70a-62bd1c037429" containerName="kube-rbac-proxy"
Sep 17 18:30:57.291169 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.291067    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11c4b90-647e-439f-a516-06810755fc93" containerName="kube-rbac-proxy"
Sep 17 18:30:57.295487 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.295470    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.298081 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.298059    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ad101d-predictor-serving-cert\""
Sep 17 18:30:57.298198 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.298084    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:30:57.298198 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.298091    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ad101d-kube-rbac-proxy-sar-config\""
Sep 17 18:30:57.298198 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.298060    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:30:57.298198 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.298187    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:30:57.298460 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.298204    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tw7lv\""
Sep 17 18:30:57.298760 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.298747    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:30:57.304427 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.304405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"]
Sep 17 18:30:57.397095 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.397068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-proxy-tls\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.397243 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.397104    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.397243 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.397126    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvz96\" (UniqueName: \"kubernetes.io/projected/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-api-access-pvz96\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.397243 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.397170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kserve-provision-location\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.397243 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.397215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-cabundle-cert\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.497974 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.497947    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-proxy-tls\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.498100 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.497985    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.498100 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.498003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvz96\" (UniqueName: \"kubernetes.io/projected/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-api-access-pvz96\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.498100 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.498022    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kserve-provision-location\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.498100 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.498040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-cabundle-cert\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.498515 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.498498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kserve-provision-location\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.498685 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.498666    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.498749 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.498709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-cabundle-cert\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.500408 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.500389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-proxy-tls\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.506851 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.506823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvz96\" (UniqueName: \"kubernetes.io/projected/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-api-access-pvz96\") pod \"isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") " pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.607057 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.607004    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:30:57.881357 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.881282    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"]
Sep 17 18:30:57.884256 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:30:57.884220    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5d4796_9388_4fe8_a495_9dc2b3628c2e.slice/crio-d355c4835eba24c8e974dffe208d9e52e68f2e5c08ad0f08b22a3905e911b7ba WatchSource:0}: Error finding container d355c4835eba24c8e974dffe208d9e52e68f2e5c08ad0f08b22a3905e911b7ba: Status 404 returned error can't find the container with id d355c4835eba24c8e974dffe208d9e52e68f2e5c08ad0f08b22a3905e911b7ba
Sep 17 18:30:57.966255 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.966228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerStarted","Data":"26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da"}
Sep 17 18:30:57.966602 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:30:57.966294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerStarted","Data":"d355c4835eba24c8e974dffe208d9e52e68f2e5c08ad0f08b22a3905e911b7ba"}
Sep 17 18:31:01.983411 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:01.983378    2562 generic.go:356] "Generic (PLEG): container finished" podID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerID="26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da" exitCode=0
Sep 17 18:31:01.983888 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:01.983456    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerDied","Data":"26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da"}
Sep 17 18:31:02.989461 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:02.989429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerStarted","Data":"41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b"}
Sep 17 18:31:02.989845 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:02.989487    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerStarted","Data":"5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989"}
Sep 17 18:31:02.989845 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:02.989812    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:31:02.989960 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:02.989855    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:31:02.991163 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:02.991139    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:31:03.008700 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:03.008653    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podStartSLOduration=2.473476141 podStartE2EDuration="6.008640065s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.535163924s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:03.007002116 +0000 UTC m=+3354.194437967" watchObservedRunningTime="2026-09-17 18:31:03.008640065 +0000 UTC m=+3354.196075914"
Sep 17 18:31:03.993537 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:03.993504    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:31:08.997808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:08.997772    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:31:08.998350 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:08.998324    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:31:18.998627 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:18.998589    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:31:28.998337 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:28.998300    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:31:38.999232 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:38.999190    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:31:48.998556 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:48.998516    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:31:58.999325 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:31:58.999244    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:32:08.999134 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:08.999093    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 18:32:18.998825 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:18.998790    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:32:42.495873 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.495840    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"]
Sep 17 18:32:42.499139 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.499123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.501400 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.501375    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ad101d-predictor-serving-cert\""
Sep 17 18:32:42.501531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.501420    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ad101d\""
Sep 17 18:32:42.501531 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.501431    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ad101d-kube-rbac-proxy-sar-config\""
Sep 17 18:32:42.501748 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.501733    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ad101d-dockercfg-klvwx\""
Sep 17 18:32:42.508658 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.508637    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"]
Sep 17 18:32:42.533129 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.533098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxhn\" (UniqueName: \"kubernetes.io/projected/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-api-access-gnxhn\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.533239 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.533132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.533239 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.533152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.533239 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.533173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kserve-provision-location\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.533239 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.533208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-cabundle-cert\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.634235 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.634209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxhn\" (UniqueName: \"kubernetes.io/projected/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-api-access-gnxhn\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.634374 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.634241    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.634374 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.634260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.634510 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:32:42.634492    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-serving-cert: secret "isvc-secondary-ad101d-predictor-serving-cert" not found
Sep 17 18:32:42.634569 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.634533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kserve-provision-location\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.634631 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:32:42.634567    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls podName:d5b71ffe-1f0e-46a2-b61f-503df3aebb96 nodeName:}" failed. No retries permitted until 2026-09-17 18:32:43.134547005 +0000 UTC m=+3454.321982841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls") pod "isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" (UID: "d5b71ffe-1f0e-46a2-b61f-503df3aebb96") : secret "isvc-secondary-ad101d-predictor-serving-cert" not found
Sep 17 18:32:42.634631 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.634603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-cabundle-cert\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.634869 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.634850    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kserve-provision-location\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.635023 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.635002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.635189 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.635175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-cabundle-cert\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:42.643424 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:42.643404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxhn\" (UniqueName: \"kubernetes.io/projected/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-api-access-gnxhn\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:43.138867 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:43.138823    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:43.141188 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:43.141161    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls\") pod \"isvc-secondary-ad101d-predictor-547b99dd4f-46hvg\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") " pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:43.409425 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:43.409347    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:32:43.683378 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:43.683289    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"]
Sep 17 18:32:43.686293 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:32:43.686263    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b71ffe_1f0e_46a2_b61f_503df3aebb96.slice/crio-f1a1d6fae56bf64b6c7a81177d1c4598aa55122d6016e2e2b46f49891d248f5d WatchSource:0}: Error finding container f1a1d6fae56bf64b6c7a81177d1c4598aa55122d6016e2e2b46f49891d248f5d: Status 404 returned error can't find the container with id f1a1d6fae56bf64b6c7a81177d1c4598aa55122d6016e2e2b46f49891d248f5d
Sep 17 18:32:44.342232 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:44.342193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" event={"ID":"d5b71ffe-1f0e-46a2-b61f-503df3aebb96","Type":"ContainerStarted","Data":"534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6"}
Sep 17 18:32:44.342411 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:44.342271    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" event={"ID":"d5b71ffe-1f0e-46a2-b61f-503df3aebb96","Type":"ContainerStarted","Data":"f1a1d6fae56bf64b6c7a81177d1c4598aa55122d6016e2e2b46f49891d248f5d"}
Sep 17 18:32:49.363127 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:49.363097    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_d5b71ffe-1f0e-46a2-b61f-503df3aebb96/storage-initializer/0.log"
Sep 17 18:32:49.363528 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:49.363133    2562 generic.go:356] "Generic (PLEG): container finished" podID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerID="534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6" exitCode=1
Sep 17 18:32:49.363528 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:49.363211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" event={"ID":"d5b71ffe-1f0e-46a2-b61f-503df3aebb96","Type":"ContainerDied","Data":"534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6"}
Sep 17 18:32:50.368966 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:50.368943    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_d5b71ffe-1f0e-46a2-b61f-503df3aebb96/storage-initializer/0.log"
Sep 17 18:32:50.369372 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:50.369014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" event={"ID":"d5b71ffe-1f0e-46a2-b61f-503df3aebb96","Type":"ContainerStarted","Data":"8342809c3c4c135c47ab3f7081f30888c8aa6bacfe7f8434459e9910660be4ac"}
Sep 17 18:32:56.392157 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:56.392126    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_d5b71ffe-1f0e-46a2-b61f-503df3aebb96/storage-initializer/1.log"
Sep 17 18:32:56.392548 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:56.392474    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_d5b71ffe-1f0e-46a2-b61f-503df3aebb96/storage-initializer/0.log"
Sep 17 18:32:56.392548 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:56.392502    2562 generic.go:356] "Generic (PLEG): container finished" podID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerID="8342809c3c4c135c47ab3f7081f30888c8aa6bacfe7f8434459e9910660be4ac" exitCode=1
Sep 17 18:32:56.392622 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:56.392568    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" event={"ID":"d5b71ffe-1f0e-46a2-b61f-503df3aebb96","Type":"ContainerDied","Data":"8342809c3c4c135c47ab3f7081f30888c8aa6bacfe7f8434459e9910660be4ac"}
Sep 17 18:32:56.392622 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:56.392618    2562 scope.go:117] "RemoveContainer" containerID="534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6"
Sep 17 18:32:56.392948 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:56.392907    2562 scope.go:117] "RemoveContainer" containerID="534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6"
Sep 17 18:32:56.402903 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:32:56.402884    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_kserve-ci-e2e-test(d5b71ffe-1f0e-46a2-b61f-503df3aebb96)\"" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96"
Sep 17 18:32:56.403272 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:32:56.403247    2562 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_kserve-ci-e2e-test_d5b71ffe-1f0e-46a2-b61f-503df3aebb96_0 in pod sandbox f1a1d6fae56bf64b6c7a81177d1c4598aa55122d6016e2e2b46f49891d248f5d from index: no such id: '534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6'" containerID="534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6"
Sep 17 18:32:56.403342 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:56.403281    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_kserve-ci-e2e-test_d5b71ffe-1f0e-46a2-b61f-503df3aebb96_0 in pod sandbox f1a1d6fae56bf64b6c7a81177d1c4598aa55122d6016e2e2b46f49891d248f5d from index: no such id: '534c5490880d2a150b9749fd3631435a46a27fe2d701c730ea94c8526ecf9ef6'"
Sep 17 18:32:57.398047 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:32:57.398022    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_d5b71ffe-1f0e-46a2-b61f-503df3aebb96/storage-initializer/1.log"
Sep 17 18:33:00.514273 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.514238    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"]
Sep 17 18:33:00.547510 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.547475    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"]
Sep 17 18:33:00.547788 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.547756    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container" containerID="cri-o://5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989" gracePeriod=30
Sep 17 18:33:00.547974 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.547769    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kube-rbac-proxy" containerID="cri-o://41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b" gracePeriod=30
Sep 17 18:33:00.665612 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.665584    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_d5b71ffe-1f0e-46a2-b61f-503df3aebb96/storage-initializer/1.log"
Sep 17 18:33:00.667699 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.667682    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:33:00.766481 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766412    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kserve-provision-location\") pod \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") "
Sep 17 18:33:00.766481 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766446    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-cabundle-cert\") pod \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") "
Sep 17 18:33:00.766639 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766502    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-rbac-proxy-sar-config\") pod \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") "
Sep 17 18:33:00.766639 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766528    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls\") pod \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") "
Sep 17 18:33:00.766639 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnxhn\" (UniqueName: \"kubernetes.io/projected/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-api-access-gnxhn\") pod \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\" (UID: \"d5b71ffe-1f0e-46a2-b61f-503df3aebb96\") "
Sep 17 18:33:00.766753 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766664    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5b71ffe-1f0e-46a2-b61f-503df3aebb96" (UID: "d5b71ffe-1f0e-46a2-b61f-503df3aebb96"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:00.766803 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766787    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:00.766838 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766813    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d5b71ffe-1f0e-46a2-b61f-503df3aebb96" (UID: "d5b71ffe-1f0e-46a2-b61f-503df3aebb96"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:00.766896 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.766871    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5b71ffe-1f0e-46a2-b61f-503df3aebb96" (UID: "d5b71ffe-1f0e-46a2-b61f-503df3aebb96"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:00.768590 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.768566    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d5b71ffe-1f0e-46a2-b61f-503df3aebb96" (UID: "d5b71ffe-1f0e-46a2-b61f-503df3aebb96"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:00.768691 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.768599    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-api-access-gnxhn" (OuterVolumeSpecName: "kube-api-access-gnxhn") pod "d5b71ffe-1f0e-46a2-b61f-503df3aebb96" (UID: "d5b71ffe-1f0e-46a2-b61f-503df3aebb96"). InnerVolumeSpecName "kube-api-access-gnxhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:00.868075 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.868044    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:00.868075 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.868068    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:00.868075 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.868078    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnxhn\" (UniqueName: \"kubernetes.io/projected/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-kube-api-access-gnxhn\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:00.868268 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:00.868087    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5b71ffe-1f0e-46a2-b61f-503df3aebb96-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:01.414318 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.414285    2562 generic.go:356] "Generic (PLEG): container finished" podID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerID="41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b" exitCode=2
Sep 17 18:33:01.414477 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.414360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerDied","Data":"41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b"}
Sep 17 18:33:01.415666 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.415648    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ad101d-predictor-547b99dd4f-46hvg_d5b71ffe-1f0e-46a2-b61f-503df3aebb96/storage-initializer/1.log"
Sep 17 18:33:01.415773 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.415701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg" event={"ID":"d5b71ffe-1f0e-46a2-b61f-503df3aebb96","Type":"ContainerDied","Data":"f1a1d6fae56bf64b6c7a81177d1c4598aa55122d6016e2e2b46f49891d248f5d"}
Sep 17 18:33:01.415773 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.415744    2562 scope.go:117] "RemoveContainer" containerID="8342809c3c4c135c47ab3f7081f30888c8aa6bacfe7f8434459e9910660be4ac"
Sep 17 18:33:01.415773 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.415749    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"
Sep 17 18:33:01.448053 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.448025    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"]
Sep 17 18:33:01.452401 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:01.452378    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ad101d-predictor-547b99dd4f-46hvg"]
Sep 17 18:33:03.276445 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:03.276412    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" path="/var/lib/kubelet/pods/d5b71ffe-1f0e-46a2-b61f-503df3aebb96/volumes"
Sep 17 18:33:03.994197 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:03.994161    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 17 18:33:05.198826 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.198801    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:33:05.301459 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvz96\" (UniqueName: \"kubernetes.io/projected/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-api-access-pvz96\") pod \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") "
Sep 17 18:33:05.301459 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301443    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-rbac-proxy-sar-config\") pod \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") "
Sep 17 18:33:05.301681 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301471    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kserve-provision-location\") pod \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") "
Sep 17 18:33:05.301681 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301499    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-proxy-tls\") pod \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") "
Sep 17 18:33:05.301681 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301556    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-cabundle-cert\") pod \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\" (UID: \"cb5d4796-9388-4fe8-a495-9dc2b3628c2e\") "
Sep 17 18:33:05.301809 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cb5d4796-9388-4fe8-a495-9dc2b3628c2e" (UID: "cb5d4796-9388-4fe8-a495-9dc2b3628c2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:05.301874 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301798    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb5d4796-9388-4fe8-a495-9dc2b3628c2e" (UID: "cb5d4796-9388-4fe8-a495-9dc2b3628c2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:05.301991 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.301971    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cb5d4796-9388-4fe8-a495-9dc2b3628c2e" (UID: "cb5d4796-9388-4fe8-a495-9dc2b3628c2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:33:05.303596 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.303576    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-api-access-pvz96" (OuterVolumeSpecName: "kube-api-access-pvz96") pod "cb5d4796-9388-4fe8-a495-9dc2b3628c2e" (UID: "cb5d4796-9388-4fe8-a495-9dc2b3628c2e"). InnerVolumeSpecName "kube-api-access-pvz96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:05.303703 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.303644    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb5d4796-9388-4fe8-a495-9dc2b3628c2e" (UID: "cb5d4796-9388-4fe8-a495-9dc2b3628c2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:05.402852 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.402821    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvz96\" (UniqueName: \"kubernetes.io/projected/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-api-access-pvz96\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:05.402852 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.402847    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:05.402852 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.402856    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-kserve-provision-location\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:05.403092 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.402866    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-proxy-tls\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:05.403092 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.402875    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb5d4796-9388-4fe8-a495-9dc2b3628c2e-cabundle-cert\") on node \"ip-10-0-132-143.ec2.internal\" DevicePath \"\""
Sep 17 18:33:05.432334 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.432303    2562 generic.go:356] "Generic (PLEG): container finished" podID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerID="5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989" exitCode=0
Sep 17 18:33:05.432483 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.432362    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"
Sep 17 18:33:05.432483 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.432355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerDied","Data":"5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989"}
Sep 17 18:33:05.432483 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.432480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4" event={"ID":"cb5d4796-9388-4fe8-a495-9dc2b3628c2e","Type":"ContainerDied","Data":"d355c4835eba24c8e974dffe208d9e52e68f2e5c08ad0f08b22a3905e911b7ba"}
Sep 17 18:33:05.432604 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.432509    2562 scope.go:117] "RemoveContainer" containerID="41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b"
Sep 17 18:33:05.440647 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.440628    2562 scope.go:117] "RemoveContainer" containerID="5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989"
Sep 17 18:33:05.447635 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.447620    2562 scope.go:117] "RemoveContainer" containerID="26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da"
Sep 17 18:33:05.454815 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.454792    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"]
Sep 17 18:33:05.454983 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.454962    2562 scope.go:117] "RemoveContainer" containerID="41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b"
Sep 17 18:33:05.455253 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:33:05.455237    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b\": container with ID starting with 41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b not found: ID does not exist" containerID="41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b"
Sep 17 18:33:05.455301 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.455265    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b"} err="failed to get container status \"41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b\": rpc error: code = NotFound desc = could not find container \"41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b\": container with ID starting with 41161fade8e7be99fb78655eb8c1436c2d8d52c08c0d941e07710155e227464b not found: ID does not exist"
Sep 17 18:33:05.455301 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.455282    2562 scope.go:117] "RemoveContainer" containerID="5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989"
Sep 17 18:33:05.455547 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:33:05.455529    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989\": container with ID starting with 5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989 not found: ID does not exist" containerID="5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989"
Sep 17 18:33:05.455598 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.455552    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989"} err="failed to get container status \"5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989\": rpc error: code = NotFound desc = could not find container \"5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989\": container with ID starting with 5a116af48c61a217be81f1df8de7e370e8c1ed25ce799485cf5b1199d6633989 not found: ID does not exist"
Sep 17 18:33:05.455598 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.455567    2562 scope.go:117] "RemoveContainer" containerID="26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da"
Sep 17 18:33:05.455790 ip-10-0-132-143 kubenswrapper[2562]: E0917 18:33:05.455775    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da\": container with ID starting with 26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da not found: ID does not exist" containerID="26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da"
Sep 17 18:33:05.455834 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.455794    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da"} err="failed to get container status \"26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da\": rpc error: code = NotFound desc = could not find container \"26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da\": container with ID starting with 26304b1e3025cbba87fadf47d9f4786f8d973306a0d0a1fc847cee547e7092da not found: ID does not exist"
Sep 17 18:33:05.459103 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:05.459077    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ad101d-predictor-75fdb7dff8-8bnz4"]
Sep 17 18:33:07.275780 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:07.275748    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" path="/var/lib/kubelet/pods/cb5d4796-9388-4fe8-a495-9dc2b3628c2e/volumes"
Sep 17 18:33:43.665431 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665361    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m4nv9/must-gather-4rq4w"]
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665604    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665616    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665640    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerName="storage-initializer"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665646    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerName="storage-initializer"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665656    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="storage-initializer"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665663    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="storage-initializer"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665669    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerName="storage-initializer"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665674    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerName="storage-initializer"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665680    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kube-rbac-proxy"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665685    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kube-rbac-proxy"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665730    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kube-rbac-proxy"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665738    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5d4796-9388-4fe8-a495-9dc2b3628c2e" containerName="kserve-container"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665744    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerName="storage-initializer"
Sep 17 18:33:43.665871 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.665749    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b71ffe-1f0e-46a2-b61f-503df3aebb96" containerName="storage-initializer"
Sep 17 18:33:43.668343 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.668326    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:43.670650 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.670615    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4nv9\"/\"openshift-service-ca.crt\""
Sep 17 18:33:43.670784 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.670685    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m4nv9\"/\"kube-root-ca.crt\""
Sep 17 18:33:43.670784 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.670776    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m4nv9\"/\"default-dockercfg-vxkht\""
Sep 17 18:33:43.676935 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.676898    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4nv9/must-gather-4rq4w"]
Sep 17 18:33:43.772400 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.772374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7df2\" (UniqueName: \"kubernetes.io/projected/176c22d8-02c3-4b78-a29c-5757ae9a8566-kube-api-access-t7df2\") pod \"must-gather-4rq4w\" (UID: \"176c22d8-02c3-4b78-a29c-5757ae9a8566\") " pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:43.772516 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.772409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/176c22d8-02c3-4b78-a29c-5757ae9a8566-must-gather-output\") pod \"must-gather-4rq4w\" (UID: \"176c22d8-02c3-4b78-a29c-5757ae9a8566\") " pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:43.873544 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.873521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7df2\" (UniqueName: \"kubernetes.io/projected/176c22d8-02c3-4b78-a29c-5757ae9a8566-kube-api-access-t7df2\") pod \"must-gather-4rq4w\" (UID: \"176c22d8-02c3-4b78-a29c-5757ae9a8566\") " pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:43.873660 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.873555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/176c22d8-02c3-4b78-a29c-5757ae9a8566-must-gather-output\") pod \"must-gather-4rq4w\" (UID: \"176c22d8-02c3-4b78-a29c-5757ae9a8566\") " pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:43.873810 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.873796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/176c22d8-02c3-4b78-a29c-5757ae9a8566-must-gather-output\") pod \"must-gather-4rq4w\" (UID: \"176c22d8-02c3-4b78-a29c-5757ae9a8566\") " pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:43.884666 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.884645    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7df2\" (UniqueName: \"kubernetes.io/projected/176c22d8-02c3-4b78-a29c-5757ae9a8566-kube-api-access-t7df2\") pod \"must-gather-4rq4w\" (UID: \"176c22d8-02c3-4b78-a29c-5757ae9a8566\") " pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:43.977414 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:43.977362    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4nv9/must-gather-4rq4w"
Sep 17 18:33:44.245967 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:44.245787    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4nv9/must-gather-4rq4w"]
Sep 17 18:33:44.248486 ip-10-0-132-143 kubenswrapper[2562]: W0917 18:33:44.248460    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176c22d8_02c3_4b78_a29c_5757ae9a8566.slice/crio-eaf148e539fc7eec3384d263e52c6a3e8d91a98865e5be7f75592934d4406b92 WatchSource:0}: Error finding container eaf148e539fc7eec3384d263e52c6a3e8d91a98865e5be7f75592934d4406b92: Status 404 returned error can't find the container with id eaf148e539fc7eec3384d263e52c6a3e8d91a98865e5be7f75592934d4406b92
Sep 17 18:33:44.250269 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:44.250253    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:33:44.570513 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:44.570425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4nv9/must-gather-4rq4w" event={"ID":"176c22d8-02c3-4b78-a29c-5757ae9a8566","Type":"ContainerStarted","Data":"eaf148e539fc7eec3384d263e52c6a3e8d91a98865e5be7f75592934d4406b92"}
Sep 17 18:33:45.576839 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:45.576804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4nv9/must-gather-4rq4w" event={"ID":"176c22d8-02c3-4b78-a29c-5757ae9a8566","Type":"ContainerStarted","Data":"5bb25b97d134b05dda1f3080892b551e6c97121bfd4a7e68053d08c98337ec5e"}
Sep 17 18:33:45.577317 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:45.576848    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4nv9/must-gather-4rq4w" event={"ID":"176c22d8-02c3-4b78-a29c-5757ae9a8566","Type":"ContainerStarted","Data":"32061180f3051c27731cdddaad8be80d10b63c9e2c2ed1d79d6b4d6cb215f916"}
Sep 17 18:33:45.594662 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:45.594610    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m4nv9/must-gather-4rq4w" podStartSLOduration=1.6159118810000002 podStartE2EDuration="2.594592592s" totalImagesPullingTime="978.680711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:45.592901086 +0000 UTC m=+3516.780336947" watchObservedRunningTime="2026-09-17 18:33:45.594592592 +0000 UTC m=+3516.782028441"
Sep 17 18:33:46.737546 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:46.737519    2562 ???:1] "http: TLS handshake error from 10.0.132.143:58142: EOF"
Sep 17 18:33:46.745138 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:46.745116    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n9dx8_8dfece24-349b-48df-84f2-5b78d5c7feb5/global-pull-secret-syncer/0.log"
Sep 17 18:33:46.969183 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:46.969137    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vbpgr_ad3032d4-e211-4afe-b702-2f06b5841986/konnectivity-agent/0.log"
Sep 17 18:33:46.991434 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:46.991359    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-143.ec2.internal_68d557cea98b36ce4a2697ffb1cea14a/haproxy/0.log"
Sep 17 18:33:51.353649 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:51.353612    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbzqn_3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac/node-exporter/0.log"
Sep 17 18:33:51.395617 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:51.395583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbzqn_3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac/kube-rbac-proxy/0.log"
Sep 17 18:33:51.436373 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:51.436225    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nbzqn_3f3efc75-ec3b-4b78-a76d-bc9c9d4c0cac/init-textfile/0.log"
Sep 17 18:33:53.443471 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:53.443443    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-hkrf6_262e63c2-cf2a-40a8-b6d0-ded6aa3978f7/networking-console-plugin/0.log"
Sep 17 18:33:54.946863 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.946830    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"]
Sep 17 18:33:54.952610 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.952584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:54.959631 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.959599    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"]
Sep 17 18:33:54.968342 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.968308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-proc\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:54.968476 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.968347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-sys\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:54.968476 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.968426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-lib-modules\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:54.968574 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.968477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-podres\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:54.968574 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:54.968500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwgp2\" (UniqueName: \"kubernetes.io/projected/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-kube-api-access-qwgp2\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.069598 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.069540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-sys\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.069808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.069611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-lib-modules\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.069808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.069689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-podres\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.069808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.069726    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwgp2\" (UniqueName: \"kubernetes.io/projected/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-kube-api-access-qwgp2\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.069808 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.069764    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-proc\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.070072 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.069871    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-proc\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.070072 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.069950    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-sys\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.070072 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.070069    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-lib-modules\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.070242 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.070143    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-podres\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.079681 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.079650    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwgp2\" (UniqueName: \"kubernetes.io/projected/ce0141e7-7687-47a6-adbe-c7a899cc1ce2-kube-api-access-qwgp2\") pod \"perf-node-gather-daemonset-84f2l\" (UID: \"ce0141e7-7687-47a6-adbe-c7a899cc1ce2\") " pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.264831 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.264790    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:55.560212 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.560132    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"]
Sep 17 18:33:55.626084 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.626056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l" event={"ID":"ce0141e7-7687-47a6-adbe-c7a899cc1ce2","Type":"ContainerStarted","Data":"11d834e75ffedb10f2c6aff4a2ddb33e2dc5262c0f56f94f4ca597b9a4dab9f6"}
Sep 17 18:33:55.938444 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.938368    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcl8d_43537637-3dfa-41b6-a106-2963abd90828/dns/0.log"
Sep 17 18:33:55.957415 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:55.957392    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rcl8d_43537637-3dfa-41b6-a106-2963abd90828/kube-rbac-proxy/0.log"
Sep 17 18:33:56.112632 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:56.112597    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-txcs7_7a655d6d-1a91-41e1-9ef1-4b3b3f7abe5f/dns-node-resolver/0.log"
Sep 17 18:33:56.632350 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:56.632313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l" event={"ID":"ce0141e7-7687-47a6-adbe-c7a899cc1ce2","Type":"ContainerStarted","Data":"e42672db4809e2e7d004195767e59a26ad3816ad477244bd593acf7c4ef76e8b"}
Sep 17 18:33:56.632581 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:56.632382    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:33:56.650200 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:56.650154    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l" podStartSLOduration=2.650139508 podStartE2EDuration="2.650139508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:56.649183311 +0000 UTC m=+3527.836619161" watchObservedRunningTime="2026-09-17 18:33:56.650139508 +0000 UTC m=+3527.837575400"
Sep 17 18:33:56.668772 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:56.668746    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z4zxd_6badc097-8f99-421a-a042-3e1991f71fc0/node-ca/0.log"
Sep 17 18:33:57.719141 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:57.719094    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qzvww_112f5254-705f-49a9-b374-1146b06e3279/serve-healthcheck-canary/0.log"
Sep 17 18:33:58.158639 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:58.158609    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvmjd_3f1fcef5-5357-408f-ae74-8e7b6ae4435b/kube-rbac-proxy/0.log"
Sep 17 18:33:58.178625 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:58.178603    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvmjd_3f1fcef5-5357-408f-ae74-8e7b6ae4435b/exporter/0.log"
Sep 17 18:33:58.198655 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:33:58.198631    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pvmjd_3f1fcef5-5357-408f-ae74-8e7b6ae4435b/extractor/0.log"
Sep 17 18:34:00.724606 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:00.724578    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-9b487cb79-phl6l_6c182504-61e0-42d0-8c08-03149103893e/manager/0.log"
Sep 17 18:34:00.787769 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:00.787740    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-26zsj_eb5dd334-765d-4c84-833a-c533ddb0005a/server/0.log"
Sep 17 18:34:01.498087 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:01.498055    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-t7rt2_a55aa06e-fcc9-430a-af47-a148b7049736/seaweedfs-tls-serving/0.log"
Sep 17 18:34:02.647590 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:02.647558    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m4nv9/perf-node-gather-daemonset-84f2l"
Sep 17 18:34:07.346674 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.346647    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w52g2_48fdef06-8806-4e8d-bc0c-794bdb713240/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:34:07.375903 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.375880    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w52g2_48fdef06-8806-4e8d-bc0c-794bdb713240/egress-router-binary-copy/0.log"
Sep 17 18:34:07.405238 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.405215    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w52g2_48fdef06-8806-4e8d-bc0c-794bdb713240/cni-plugins/0.log"
Sep 17 18:34:07.434552 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.434527    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w52g2_48fdef06-8806-4e8d-bc0c-794bdb713240/bond-cni-plugin/0.log"
Sep 17 18:34:07.468072 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.468046    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w52g2_48fdef06-8806-4e8d-bc0c-794bdb713240/routeoverride-cni/0.log"
Sep 17 18:34:07.506242 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.506215    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w52g2_48fdef06-8806-4e8d-bc0c-794bdb713240/whereabouts-cni-bincopy/0.log"
Sep 17 18:34:07.536804 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.536774    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w52g2_48fdef06-8806-4e8d-bc0c-794bdb713240/whereabouts-cni/0.log"
Sep 17 18:34:07.626609 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.626541    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nwhfq_a927a7c0-1fd8-4c09-ba41-c5bea5579bd0/kube-multus/0.log"
Sep 17 18:34:07.686615 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.686588    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-95wb9_60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6/network-metrics-daemon/0.log"
Sep 17 18:34:07.708043 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:07.708009    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-95wb9_60f7f9e4-46c8-4cf4-9cc7-5c8211c27ae6/kube-rbac-proxy/0.log"
Sep 17 18:34:09.333201 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.333172    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/ovn-controller/0.log"
Sep 17 18:34:09.352390 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.352343    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/ovn-acl-logging/0.log"
Sep 17 18:34:09.377881 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.377852    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/kube-rbac-proxy-node/0.log"
Sep 17 18:34:09.398379 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.398353    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:34:09.417655 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.417628    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/northd/0.log"
Sep 17 18:34:09.440424 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.440395    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/nbdb/0.log"
Sep 17 18:34:09.472326 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.472291    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/sbdb/0.log"
Sep 17 18:34:09.523526 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:09.523495    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qkrtg_20234e60-7bf9-440e-8c44-7006a3c67a8f/ovnkube-controller/0.log"
Sep 17 18:34:10.596842 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:10.596810    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dk7v5_9bba983c-c289-4cbe-a15f-5e3877032bdd/network-check-target-container/0.log"
Sep 17 18:34:11.694203 ip-10-0-132-143 kubenswrapper[2562]: I0917 18:34:11.694110    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qfld2_4f32820d-0d24-47a3-a833-8701f0844ab6/iptables-alerter/0.log"